Intro
There's a pitch behind every "AI writes your tests too" workflow: more coverage, less manual toil, a safety net that used to take a sprint now takes minutes.
The pitch skips over what that safety net is actually made of. When the same model writes the implementation and the test suite, you haven't added a second, independent check. You've asked one reviewer to grade its own homework and handed you the green checkmark as if someone else had signed off.
The blind spot loop
A model reasons about a function once, forms an implicit set of assumptions (input shapes, timezone handling, what counts as "empty"), and writes the implementation against those assumptions. Ask the same model to write tests for that function, and it doesn't re-derive correct behavior from scratch. It writes tests against the same assumptions it just used to write the code. If it assumed dates always arrive as ISO strings in UTC, the implementation assumes that, and the tests assume it too. The suite goes green. The assumption is still wrong.
Tests that pass for the wrong reason
(Illustrative, not a specific case, but recognizable to anyone who's shipped an AI-generated suite.) Picture a discount-calculation function where the model assumes quantities are always positive integers. The implementation skips a negative-quantity check. The generated tests exercise 1, 5, and 100, because those are the "normal" values a model reaching for plausible test data will reach for. Nothing ever asks what happens at -1 or 0, because neither pass, the code or the tests, ever considered them worth asking about. Coverage tooling reports 100% on this function. The bug ships anyway.
Coverage becomes a false signal
High line or branch coverage from an AI-authored suite tells you the code paths were exercised, not that the right inputs exercised them. A suite can hit every line of a function and still never send it a null, an empty array, a duplicate key, or a value at a type boundary, if the author, human or model, never imagined those as possibilities in the first place. Coverage percentage was never built to detect a shared blind spot. It just counts what got tried.
What to keep human, and what to hand to AI
The fix isn't "stop using AI for tests." It's separating the two jobs testing actually does:
- Defining correct behavior (the assertions) should come from a human who understands the spec or the business rule the function is supposed to honor, decided independently of however the implementation happens to work.
- Generating volume and variety (mock data, randomized inputs, edge-case permutations) is exactly what AI is good at, and doing it well doesn't require the model to have written the implementation.
- Human review time is best spent on boundary conditions specifically: zero, negative, empty, duplicate, malformed, concurrent, because that's statistically where both human and AI implementations tend to fail first.
This isn't really a testing problem. It's a correlated-error problem wearing a green checkmark. Two independent reviewers catch different mistakes because they're independent. One reviewer checking its own work twice catches the same mistakes it already missed, twice.
Where's your line? Do you let AI touch your test assertions at all, or only the scaffolding, mocks, fixtures, input generation, around them?
Top comments (17)
The cheapest defence I know against this costs about a minute, and I only started using it after being caught by exactly your loop.
I wrote a function that splits work across streams, and a test for it. Both green. Then, on a whim, I replaced the algorithm with a deliberately wrong one - plain block-splitting instead of the interleaving it was supposed to do. The test stayed green. It had rebuilt the same computation to check the computation, so it agreed with whatever the implementation did. Written by one mind from one understanding, exactly as you describe, and the fact that the mind was mine rather than a model's changed nothing.
So the rule here now is: after writing a guard, break the thing it guards and watch it go red. A test nobody has ever seen fail is not evidence, whoever wrote it. It also picks the fight your coverage number cannot - coverage tells you the line ran, the counter-check tells you the assertion bites.
On your question, my line ended up somewhere slightly different from where you drew it. It is not about which side writes the assertion. It is that unit tests only ever prove the code does what its author meant. Last night I wrote a detector plus its tests, all green, and then ran it against real stored data: it found nothing, because the keys I had chosen were not the kind of key that can collide. No test would have caught that. Only the data did.
Mutation testing by hand, basically, and yeah it's the cheapest reality check going. I've started doing something similar: after a test passes I go break the invariant it claims to enforce and make sure it actually screams. If it doesn't, the assertion was decorative. Cheap, fast, embarrassing how often it catches something.
Your second point is the one that really lands though. My whole post was about correlated authorship, but you're pointing at a bigger version: correlated worldview. The author (human or model) picks the test inputs from the same mental model that shaped the code, so the tests can only probe the space the author already imagined. Real data doesn't care what you imagined. I think that's why staging replays and shadow traffic end up finding stuff no suite ever does, the inputs weren't curated by anyone with a stake in the code being right.
"Real data doesn't care what you imagined" holds one storey up too, and it caught me the same week.
I built a detector for a specific defect and measured it on eleven known-good pairs: median score 0.966, eight of eleven above 0.9. On that table alone I would have shipped it. Then I ran the same thing on pairs with no relation at all - the control I nearly skipped, because the first number already looked settled. Median 0.818, five of eleven above 0.9. At the threshold that made the first table look good, precision was 62 %.
Nothing about the eleven real pairs was wrong. I had picked which half of the data to look at, out of the same mental model that built the detector. Your loop, one level up: the measurement inherits the blind spot as readily as the test does.
The counter-check at that level is naming the number before you run it. A different detector I tried the next day found nothing at all in 735 real entries - and "nothing" is the dangerous shape, because it reads as "there was nothing to find" rather than "this design cannot fire". It was the second one. Writing "≥ 5 of 11 known cases, at most 5 false positives" beforehand turns a zero into a refutation instead of a shrug.
That is also the gap your replay argument does not close, and it is worth naming next to it: shadow traffic surfaces inputs you did not imagine, but it cannot surface an input that never occurs. A null result nobody predicted in advance is invisible to both a suite and a replay.
Yeah, this is the version I should have written. You've basically got three nested layers of the same failure: the code inherits the author's model, the tests inherit the code's model, and the evaluation inherits the author's model of what's worth measuring. Replay fixes layer two, sometimes. It does nothing for layer three because the metric itself is still curated by the person who wants the thing to work. Selection of the eval set is doing the same job selection of test inputs did one floor down.
Pre-registering the number is the right move, and it's underused outside of research contexts. ≥ 5 of 11, at most 5 FPs is a claim that can lose. "Median 0.966 on known-goods" is a claim that already won by the time you wrote it down. And your null-result point is the sharpest part: a silent detector is indistinguishable from a well-behaved world unless you said in advance what a firing one should have looked like. I'm going to steal that framing, "a zero is only a refutation if someone predicted a non-zero," for the follow-up.
Three layers is the right count, and there is a fourth underneath it that I only found by accident last night: the assumption that one run is a measurement.
A restart made me run the same condition three times with identical inputs - same tasks, same harness, and a control arm with no memory at all. It solved 15, 15 and 19 of 30. Four cells of movement with nothing changed.
Which means the pre-registered number needs a companion: how big is the number that would have shown up anyway? The cheap version is running the control twice before believing a single delta. One extra run, and it tells you whether you have a result or a coin.
Right, and that's the layer nobody wants to find because it makes every previous number provisional. If your control moves four points between identical runs, then any effect smaller than the run-to-run spread was never an effect, it was weather. And the worst part is you don't get to know the spread from the run that produced your headline number. It has to come from the runs you almost didn't do.
Two controls before you believe a delta is a good default. The uncomfortable corollary is that a lot of published-looking single-run comparisons in this space are reporting noise with a narrative attached. "N=1 with vibes" basically. I like that your accident forced the issue, mine usually come from the same place, some infrastructure hiccup makes me rerun something I thought was settled and the second number disagrees with the first loudly enough that I can't unsee it.
The most dangerous variant of this happens with mock setup. When the model authors both sides, it constructs mocks that mirror the exact assumptions of its own internal call chain. The test passes with full line coverage, but the moment you run against a real database or an actual network payload with null fields or unexpected casing, the whole thing falls over immediately.
In my agent workflows, the only reliable way to break that self-grading loop is property-based testing and mutation checks. If a generated test suite still passes when an AST mutator flips comparison operators or drops error branches, the suite gets rejected before the patch even lands in git.
The "correlated-error problem wearing a green checkmark" is the sharpest phrasing I've seen for this. Asking the model to author both the implementation and the test suite is essentially asking an assumption to grade itself.
Where we draw the line in our workflows: AI can generate candidate inputs, fuzz fixtures, and property permutations, but assertions must be deterministic invariants authored independently.
One safeguard that has saved us repeatedly is running mutation testing (or intentional fault injection) against AI-authored suites before trusting them. If you deliberately inject a defect into the function—flip an operator, drop a boundary check, return an empty set—and the AI-generated test suite still reports green, you've proved that the suite is testing plausible appearance rather than actual behavior.
If a test doesn't fail when the code is intentionally broken, its green checkmark is worse than no test at all because it launders false confidence into the PR.
"Assertions must be deterministic invariants authored independently" is a cleaner phrasing than mine and I'm probably going to end up using it. The independence has to attach to the invariant, not to the keystrokes, which is what I fumbled in the post by framing it as who-types-what.
The mutation testing point came up earlier in the thread too and I'm increasingly convinced it's the load-bearing practice here, not a nice-to-have. A suite that has never gone red under a deliberate break is an unfalsified claim, not a passing one, and those two things look identical in CI. Your last line is the part I want to sit with: a green test that can't fail is strictly worse than no test, because a missing test is a known gap and a lying test is a gap dressed as coverage. The PR reviewer downstream has no way to tell them apart from the checkmark alone.
The discount example puts the blind spot somewhere the fix does not reach. The assertion for quantity 5 was right; what never happened is that anyone asked about -1 or 0, and by the article's own account 1, 5 and 100 are what a model reaching for plausible test data reaches for. A second model that never saw the implementation reaches for the same three, so the second fix point holds and the list is unchanged. The third point does put a human on zero and negative, but as review downstream of a generated suite, so the human is left noticing which classes the list never touched, which is the review that fails quietly.
So I would move the human one step earlier, on a different axis than assertions versus scaffolding. Before anything is generated, a human takes the input classes from the spec, not the code, can quantity be zero, negative, fractional, absent, and hands those classes to generation. Inside a named class the model can then write both the values and the assertions, which is where this parts ways with the first point: the correlated thing was never who wrote the assertion, it was whose idea of normal decided which inputs exist, and once the classes come from the spec that idea no longer chooses. The thread already widened this to worldview; naming classes upstream is where a worldview gets written down instead of hunted for afterwards.
The check that follows is mechanical: take the boundary classes the spec names, and for each one ask whether any input in the generated suite falls inside it. The classes with nothing in them are the ones nobody's idea of normal had a name for, and that is a list you can read rather than a gap you have to hunt for.
You're right that the specific fix I wrote collapses under its own example. Second model reaches for the same 1, 5, 100 because "plausible test data" is a shared prior, not an independent one, and the human-on-boundaries step happens too late to change what got generated. The correlated thing isn't the assertion, it's the input taxonomy, and I had the human enter the room after the taxonomy was already implicit.
Naming classes from the spec upstream is the right relocation. It also gives you the mechanical check almost for free: class-by-class occupancy is a list you can literally print, and an empty row is a specific, nameable gap rather than a "did we think of everything" feeling. The part I'd add, and it's more your framing than mine, is that the spec itself is another worldview, so the classes you write down are only as good as the spec's imagination of its own inputs. Fractional quantities only appear as a class if someone once cared enough to say so. Which is maybe fine, the point isn't that the list is complete, it's that it exists as a written object something can be checked against, instead of living in whoever-wrote-it-last's head.
That PR reviewer blind spot is exactly where the risk compounds in automated pipelines. When a human or CI checks a green build, the checkmark only proves the runner executed to completion without throwing—not that the assertions actually constrain the system's behavior.
The only reliable countermeasure we've found in agent workflows is treating mutation kills as a required verification gate on new test files: injecting deliberate semantic mutants into the touched AST and verifying the new assertions actually turn red. If the suite survives the mutated invariant, the test is tautological regardless of line coverage.
Curious how you handle the runtime cost of mutation passes on larger repos at Cyclopt—do you scope the mutations strictly to the git diff / affected AST slice to keep CI feedback loops fast, or do you run broader sweeps asynchronously?
Counterintuitive take: AI-generated tests don't just test code; they test our assumptions. If they pass, it might mean the AI learned our biases, not that the app is bug-free. Humans still need to write the real, messy tests.
This matches what I keep seeing while running evals on my own agent's memory system. When the same process writes both the implementation and the tests, the suite is really a consistency check — it verifies the code agrees with its own assumptions, not that the assumptions are right. The nastiest variant I've hit: eval cases derived from the same examples the system was built against, so a 100% pass rate just means "it recognizes what it was fed." Two practices that helped me: (1) write the assertion table from the spec before looking at the implementation — the independent oracle is the hard part, generating input volume is the easy part; (2) don't over-trust small negative sets — 8 edge cases with zero failures still leaves a ~30% upper bound on the false-positive rate, so "we tested the boundaries" needs sample sizes to mean anything. Coverage counts what got exercised; only an oracle that never saw the implementation can tell you whether it passed for the right reason.
Very good point, I went through this and published a bunch of articles. What has helped me
Usually when some scenarios keep falling and LLM cannot fix, I bring in the expensive LLM to review and oftben it helps me close the gap quickly but it can cost you tokens
I do save all learnings which I use later to write articles and I use learnings from related projects when I plan or write other code
Spot on. AI-generated tests pass because they test what the AI knows, not what your code does. I now require human-written regression cases for every production failure before the AI can generate new tests.
the blind spot loop is exactly what we hit with LLM agent evals. the model generating test trajectories has the same implicit assumptions as the agent itself: tool call ordering, valid input shapes, what "done" looks like. eval harness goes green. agent is still wrong on edge inputs nobody imagined.
the fix that worked: spec assertions in prose first, then let the model generate parameterized inputs. keeps behavior definition out of the model's hands.
how do you handle the case where the spec itself is ambiguous — does test generation surface that, or does the model just pick an interpretation silently?