Everything in this book follows from four ideas. They are not opinions about architecture. They are what is left after the outages.
1. A model is an unreliable network call wearing a costume.
It is a third-party HTTP endpoint with high latency, variable output, occasional 429s and a bill attached. Every instinct you have about integrating a flaky payment gateway is correct here. Timeout it. Cap it. Retry the failures worth retrying and none of the others. Assume it fails, and decide now what the user sees when it does.
2. Prompts are code.
A prompt determines behaviour. That makes it code, and code belongs in version control, in review and in a deploy: not in a database row somebody edits on a Friday, and not in a config file nobody diffs. When a prompt changes, behaviour changes, and you should be able to point at the commit.
3. Cap everything: tokens, wall-clock, iterations, money.
An uncapped AI feature is an uncapped invoice. Every loop needs a maximum number of steps. Every job needs a deadline shorter than its timeout. Every tenant needs a ceiling. Every response needs a token limit chosen by you on the server, not implied by the client and not inherited from a package default. You will not notice the missing cap until the month you do.
4. The model is an untrusted caller.
It can be persuaded. It will be persuaded. Anything you enforce by asking the model nicely is not enforced. Verification, ownership, quotas and permissions live in your database and your middleware, behind the tools, never in the instructions.
How I Learned This
The first version of our agent worked on the first try, which was the problem.
It was a chat box. You told it what your business was, and it built you a website. In the demo it was genuinely delightful. It asked good questions, it made reasonable choices, and about a minute later there was a real site at a real address. We showed it to everyone.
The first thing that broke was time. A turn where the model used several tools took well over a minute, and the whole thing ran inside the HTTP request. Users watched a spinner until something upstream gave up, and then saw nothing at all: no error, no partial answer, no site, although we had already paid for every token. The work had been done. We just had nowhere to put it.
So we moved it to the queue, and the second thing broke. The job had a timeout, and a long turn hit it. The job was killed mid-turn, having written nothing, while the widget politely polled a cache key that would never appear. The user sat in front of a chat that had simply stopped talking.
Then a turn came back truncated. We had capped output tokens, sensibly, but we never looked at why the model stopped. A cut-off reply fell through the same branch as a finished one and rendered as an answer. It read fine. It was wrong.
Then a user pressed the button twice.
The tool that created a site had been told, in its own description, to only ever do this once. That instruction was honoured perfectly, right up until two requests arrived close enough together that “once” was ambiguous. We got two provisioning runs for one person. The fix was not a better instruction. The fix was a unique index.
And then, months later, we took the irreversible step away from the model altogether. Today the agent’s last tool does not build anything. It hands the person a checkout link, and the site is built only when a human puts a card on file. The model can no longer spend our money, because no instruction we could write was ever going to be as strong as a door it cannot open.
None of these were problems with the model. The model was fine. Every failure was in the code around it, and every fix was something I already knew how to do. I had just never thought to do it here, because this part had a costume on.
That is why the chapters in this book look the way they do. They are not about prompting. They are about queues, deadlines, unique indexes, trust boundaries and structured logs, applied to a component that happens to be non-deterministic.