There’s a sketch I keep coming back to: an iceberg. Above the waterline, what QA sees — bugs, a broken process, bad architecture, a testing gap, wasted engineering time. Below it, what the problem is actually attached to — someone’s decision, someone’s budget, someone’s reputation, someone’s promotion, years of defending “why we do it this way.”

That was the hard part of the job for a decade. Not finding the bug. Surviving the politics of the bug.

Then AI arrived, and something strange happened. The iceberg didn’t get smaller. It got emptier.

The accountability vacuum

Ask who owns a defect now and you get a shrug. The story was drafted by AI from a two-line prompt. The code was generated by AI from the story. The tests were generated by AI from the same story. The pull request was reviewed by AI, approved by a human who scrolled to the bottom in four seconds because there were nine hundred lines and eleven other PRs waiting.

Nobody defends this code, because nobody wrote it.

We used to complain that the political layer of the iceberg made bugs hard to fix. We were wrong about what it meant. The politics were painful, but they were evidence of ownership. Someone fought for that design. Someone’s name was on it. You could find that person, argue with them, and eventually change their mind — and when they changed their mind, the system actually changed.

Now the fight is gone and so is the leverage. “The AI wrote it” is a perfect alibi. It absorbs blame without ever learning anything.

But the alibi doesn’t survive a second question: who orchestrated the AI? Someone wrote the two-line prompt. Someone accepted the story without checking it against a real user. Someone ran the pipeline, and someone clicked merge. Every one of those is a human decision, and none of them stopped being decisions because a model executed them — an author who dictates to a typist still wrote the book. The vacuum isn’t real. It’s unclaimed. And unclaimed accountability is something a team ends up with by default, never something it chooses on purpose.

The closed loop

Here’s the part that should worry us more than accountability, because it’s structural rather than cultural.

A test is only meaningful if it comes from a source of truth that is independent of the thing being tested. That’s the whole idea. Someone decides what correct behavior is; something else checks whether the system does it.

Look at the chain we’ve built:

• The story is generated from a prompt nobody validated.
• The code is generated from the story.
• The tests are generated from the story.
• The review is generated by the same class of model that generated the code.

Every artifact in that chain derives from the same unexamined assumption. So when the pipeline goes green, what has it actually proven? Internal consistency. Nothing else. The code does what the story said, and the story might have been an invention — a plausible-sounding paragraph that has never been checked against a real user, a real regulation, or a real business rule.

We are not testing whether the software is right. We are testing whether the software agrees with itself. And it always will.

This is why the metrics look so good and the escaped defects don’t improve. Coverage is up. Test count has tripled. Cycle time is down. And production keeps surprising us, because coverage of the wrong specification is not quality — it’s confidence with better formatting.

I made a version of this argument in an earlier post about the gap between testing and quality assurance. This is that same gap one level upstream: not just that AI tests shallowly, but that the artifact setting the depth was never independent in the first place.

Yes, this is checkbox testing

Let’s stop being polite about it. If QA’s contribution is that a suite of AI-generated tests ran against AI-generated code implementing an AI-generated story, then QA has been reduced to a green tick in a pipeline. That’s not testing. That’s ceremony.

The uncomfortable truth is that AI is exceptionally good at producing the artifacts of quality and no better than us at producing quality. It generates test cases, coverage reports, traceability matrices, risk registers, sign-off documents. All the paperwork of assurance, at scale, in seconds. In a regulated industry this is genuinely dangerous, because the paperwork is what auditors see and the paperwork now looks flawless.

Picture the FDA audit. “Who authored this validation evidence, and against what requirement?” The honest answer is a model, working from a prompt nobody kept. In a domain where the trace is the point — where validation means demonstrating that a specific human verified a specific intended use — flawless paperwork with nobody behind it isn’t a compliance shortcut. It’s the finding.

Comprehension is the bottleneck now

The other shift that gets missed is where the constraint sits. Producing code used to be the expensive part, and review was cheap by comparison. Now production is nearly free and comprehension is the scarce resource. Quality is no longer limited by how fast we can build — it’s limited by how much a human can actually hold in their head. If your team generates more output per day than your humans can meaningfully read, you don’t have a productivity gain. You have an unreviewed codebase with good velocity charts.

So what is the human actually for?

“Humans will orchestrate” is the comfortable answer, and it’s half a sentence short of being useful. Orchestrate what, exactly?

The one thing AI structurally cannot do is decide what correct means. Testers have a name for this — the oracle problem. A model can tell you what the code does. It cannot tell you what it should do, because “should” comes from outside the software: from a clinician’s workflow, a patient’s confusion at 2am, a regulator’s intent, a contract, a consequence someone will live with.

That’s the job now. Not writing checks — owning the oracle.

Concretely, that means:

Break the closed loop. Tests must not derive from the same artifact as the code. Human-authored acceptance criteria, written before generation, become the ground truth. If a human didn’t specify it, an AI-written test proves nothing about it.

Keep provenance. Every requirement should carry a name and a reason. Who decided this? Against what evidence? The iceberg’s lower half was ugly, but it was traceable. Reintroduce the trace on purpose.

Put a name on the merge. Not a rubber stamp — an actual claim: I read this, I know what it’s for, and I’ll own the incident. If nobody on the team is willing to make that claim about a change, that refusal is the signal. The change waits.

Cap generation at review capacity. If we can’t read it, we don’t merge it. Generating less is a legitimate quality control.

Sample adversarially. Don’t review AI output uniformly. Pick the highest-consequence paths and go through them line by line, by hand, with hostility.

Change what you measure. Coverage and test counts are now trivially gameable by a machine. Measure escaped defects, and especially intent defects — the ones where the code worked exactly as specified and the specification was wrong. The dose reminder that fired at 9am server time instead of the patient’s time zone is an intent defect: every test passed, because the story said 9am and nobody who had talked to a patient wrote the story. That class of bug is invisible to generated testing, which is exactly why its count is the real report card. Same instinct as the mutation and negative-testing instruments I laid out previously — stop trusting a metric a machine can satisfy without understanding it.

Do the unglamorous human things. Talk to the users. Sit with support tickets. Ask the person who has been here six years why this module is fragile. None of this is in the model’s context window.

And if tomorrow there’s no AI?

Ask it literally first — outage, price change, vendor lock-in, an air-gapped client, a regulator who rules generated evidence inadmissible. Answer honestly, write it down.

Then ask the version that actually keeps me up. What happens to a QA engineer who has never designed a test from first principles? Who has never read a specification and felt that itch that something in paragraph four contradicts paragraph nine?

Those are muscles. They atrophy quietly, and you don’t discover they’re gone on a normal Tuesday — you discover it during an incident, when the model is confidently wrong and you have no independent way to know.

So keep a manual practice, as insurance rather than nostalgia. Write the risk analysis by hand first and let AI expand it, never originate it. Adopt one rule with teeth: anyone on the team must be able to explain, out loud, what a test is for and what its failure would mean — if nobody can, delete it, it was decoration. And train juniors on judgment, not prompts. Prompts change every six months; judgment compounds for thirty years.

The iceberg is still there

The bug is still attached to something. It always was.

The difference is that it used to be attached to a person with a budget and a reputation, and now it’s attached to a prompt nobody kept, a story nobody validated, and a review nobody read. The politics got easier. The epistemics got much, much worse.

Three things I’d take away from this:

“The AI wrote it” is an alibi, not an answer — accountability belongs to whoever orchestrated the AI. A human chose the prompt, ran the pipeline, and clicked merge, and that human owns the output the same way an author owns a book their typist produced. Quality assurance now means keeping that person in the loop by name — deliberately, as the accountable owner of what ships, not as a formality.

A pipeline of AI-generated story, code, tests, and review only proves the system agrees with itself. Coverage, test count, and green CI stop being quality signals the moment every artifact in the chain traces back to the same unvalidated assumption. They measure agreement, not correctness.

The job shifted from writing checks to owning the oracle. AI can tell you what the code does; it cannot tell you what it should do, because “should” lives outside the software, in a workflow, a regulation, a consequence someone else will live with. That’s the one part of QA that doesn’t automate.

Speed isn’t the enemy — unsigned work is. Somewhere in the chain, a human being has to look at the thing and say: I have understood this, I know what it’s supposed to do, and I am willing to put my name on it. That signature is the entire job, and it’s the one part we can’t generate.