Living on the bleeding edge of technology is exhilarating, but it can also be exhausting. The rapid pace of new models has us trying to decide which is best for any job. To make my life easier, I switched to a different question, and that move has paid dividends.
Instead of trying to judge if a model is good enough, I instead focus on how I can verify what it produces.
I look at model output in terms of a pipeline of intermediate artifacts. This "verification surface" lets me make hard or soft guarantees about the work as it progresses. However, and perhaps most importantly, it also lets me make guarantees about the final product of the entire pipeline.
My focus is on coding, but the technique is generally applicable, so I wrote it up and applied it across three very different domains:
- coding agents
- medical image segmentation
- language-model verification
The results were consistent across domains: it wasn't the model that mattered most, it was the surface used to verify the artifacts. The harness matters more than the model. Models wobble, so reliability actually lives in the systems around them.
If like me you're building agents you can't fully trust, perhaps you'll find my techniques useful.
The three papers, with the data and the framework →