Having AI Adversarially Review Its Own Code
Certainty: It genuinely worked this time, though the sample is still small
The thing I’ve grown warier of while writing code with AI: the risk isn’t that it can’t produce the code—it’s that it produces something that looks right but hides an edge-case bug, and I can’t catch it in the moment. While building this site I tried a more ruthless process:
- Hunt along separate dimensions. Have several AIs each do a pass from one specific angle—correctness, accessibility, interaction, consistency—instead of a vague “take a look and see if anything’s wrong.”
- Verify adversarially. For every issue that gets flagged, send in another AI whose only job is to rebut it, defaulting to “not a real problem” whenever it’s unsure. Only the findings that survive the rebuttal count.
Running this across two sites, I confirmed and fixed more than thirty genuine problems—and the nastiest of them (the slug algorithm implemented three inconsistent ways) would have been very hard to catch by eye on my own.
What makes it work is splitting “generation” and “critique” into two independent roles. That dovetails with a call I’d made earlier: when pairing with AI, the ability to review is worth more than the ability to code—and now even the reviewing can be handed off to AI. But designing the process, and making the final call on what to trust, is still on me.
Belongs to: Building My Own Sites, AI Coding Workflow