AI failures are inevitable. Whether they propagate is a design choice.
Let me give an analogy. Imagine three robots baking a cake. One shops for ingredients, one measures, and one combines and cooks them.
Robot 1 misreads "sugar" as "salt." Robot 2 faithfully measures out a cup of salt. Robot 3 bakes it into the cake.
The cake is inedible. Robots 2 and 3 did their jobs perfectly. Robot 1 made a mistake, but the real problem is that nobody caught it before the salt reached the mixing bowl.
Now add a simple checkpoint between Robot 1 and Robot 2: does the ingredient match the recipe? That's it. One checklist. The salt never reaches the mixing bowl.
This also explains why long-running AI agents drift. Every step builds on the last one. A small mistake early on gets magnified as downstream stages make perfectly "reasonable" decisions based on a flawed handoff. Course-correct along the way and the drift largely disappears.
That's how agent pipelines fail in practice
I ran autonomous AI agents for 97 days shipping production code and logged over 5,000 quality checks. The data showed that 87% of errors were predictable: things left out or consistently wrong in the same way, not wild hallucinations.
The pattern was always the same. A mistake at one stage flows forward, and every stage after it faithfully builds on the wrong thing. The AI isn't hallucinating. It's doing exactly what you'd expect given what it was handed.
The fix is the same as with the robots: don't start by trying to build "better AI." Build better checkpoints between stages. Check every handoff. The arrangement of checkpoints determines reliability more than the capability of any single model.
Three principles
- You don't need better robots. You need checkpoints between them.
- Errors compound forward. Catch them early or pay for them at every stage.
- Reliability is a property of the system, not the model.
That's the core idea behind Trust Topology, a framework I've been developing from this data.
Read the full Trust Topology writeup →