Coding agents are trained on common patterns, and you see that emerge in the code they produce. Even when that pattern has a giant security vulnerability.
I had a recent discussion on Reddit where someone found that their LLM-generated code was using user-provided metadata to assign a security role. The impact was severe: any user could simply manually edit the metadata they provided to mark themselves as admin, and it would dutifully grant it.
The code compiled just fine, all the unit tests passed, and manual QA looked clean. The code did exactly what you'd expect from a plain reading, and it was wrong.
The thing that stuck with me was that none of my normal checks (honestly, even human-manual ones) were designed to catch this. The acceptance criteria say "can a user log in" not "a user cannot manually promote themselves".
The flaw isn't the code, it's what's being checked. A check can only catch what it can see, and most checks see far less than we assume.
I call this the verification surface: the set of things a given check can actually see.
A test that runs the app sees crashes. It can't see privilege escalation. Those are different surfaces, and "it passed" only means "it passed through the surfaces you happened to check". That's why I don't think reliability is about the model. A smarter model writes the same plausible-but-flawed code. Reliability comes from checking against a surface that can see the failure you actually care about.
You can only trust what you can verify. The catch is that what you can verify is the surface, and most of us never chose it on purpose.
The framework behind this: Trust Topology →