Ask a booking system whether you can bring your nephew on Sunday, and the failure that arrives first is not the model's answer. It is that the paragraph about weekends was never put in front of it.
Ayşe Tunç maintains the booking software for four climbing walls around Manchester, and the feature asked for was small: a member types a question in ordinary words and the system answers from the house rules and the timetable. The model was the part that already worked. Everything that went wrong for a month was on either side of it.
The right paragraph, or nothing
The nephew question fetched the passage on membership categories, because the member had written nephew and Sunday and the passage that actually governs it says persons under eighteen and at weekends, sharing not one word with the question. The model then wrote a fluent, courteous and wrong paragraph out of the passage it was handed, which is the only thing it could have done with it.
If the right passage is not in the top few results, no model can rescue the answer. That is the sentence to keep, because it moves the work: retrieval is judged by whether the correct passage is in the handful you pass along, and that is a search problem with decades of technique behind it, none of which is about models. Ayşe fixed it by indexing each rule twice, once as written and once as the question it answers, which is a piece of writing rather than a piece of engineering.
Fine tuning teaches a shape, not a fact
Before that, she spent a fortnight fine tuning on 300 past support emails, on the reasonable theory that the system should learn how the walls actually answer. It worked. The replies came back sounding exactly like the front desk, in the house cadence, with the house sign off, and still wrong about Sundays.
Fine tuning teaches a shape, not a fact. It moves the style, the format and the kind of thing produced, which is worth having when the output has to look a particular way every time. It is the wrong instrument for getting a specific paragraph right, and the failure it produces is the expensive kind, because the answers get more convincing while staying just as wrong. A fact that changes weekly belongs in something read at the moment of answering, never in the weights.
The eval set that told her nothing
She had written an eval set, which was the one thing done in the right order. Write the eval set before the feature, because a demo cannot fail: a demo is a question chosen after the system existed, and it selects for what already works, every time, without anybody deciding to cheat.
The first version still taught her nothing. Twelve questions, all of them things a member plausibly asks, all of them answered correctly on the first afternoon. An eval with no wrong answers in it measures only that the thing runs. The set that earned its keep was the second one, built from the questions the system had got wrong that month, with the nephew question first and the expected answer written out by a person: no under eighteens on Saturday or Sunday, and the rule quoted rather than paraphrased.
Which is the shape of the whole discipline below. A system that does not answer the same way twice cannot be debugged by looking at it, so tests for a system that does not answer the same way twice is not the tidy last step, it is the instrument every other step is read through.
The two sides of the call are both yours
Every call is a network call, so it fails the way network calls fail: slowly, intermittently, and at the worst moment. It also bills. A retry on a timeout can charge twice and answer once, and a retry on a request that already booked a slot is a duplicate booking rather than a duplicate paragraph.
On the way out, treat the model's output like a form a stranger filled in. It is not markup you wrote and it is not a value from your own database; it is text produced under influences you cannot fully enumerate, and it is about to be rendered on a page, put in a query, or handed to a booking function. Parse it against a declared type, reject what does not fit, and keep the rejection, because the rejects are next month's eval cases.
Where the work is
Everything narrows to one request, and the request is the part you did not write
Four things are squeezed into a single call, and three consequences fall out of it. The call itself is a few lines.
- The member's questionTyped the way a person types on a phone
- The house rulesNine pages, written for a noticeboard
- The timetableChanging weekly, and living in another system
- The standing instructionWho the answer is for, and what may not be invented
- A parsed answerOr a rejection, kept for next month's eval set
- A booking, or a refusalA row in a database that is awkward to take back
- Somebody's SundayWhere a paragraph nobody retrieved finally arrives
The two cones are the system and the waist is the vendor. A drawing that made the waist the biggest part would be a picture of the demo rather than of the thing in service.
The order, and the step everyone skips
Make the call first, at the request, the response and the parts that surprise you, which takes an evening and settles what the thing costs and how it fails. Write the eval set second. Third, giving the model your data at the moment it answers, which is where most of the work turns out to be. Fourth and rarely, teaching a model a style or a task. Then everything between a demo and a service.
The second step is the one that gets skipped, and skipping it is not visible for about six weeks. Without it, step three cannot be debugged, because nothing distinguishes a retrieval that improved from one that changed. Step four cannot be justified at all, since the only claim fine tuning ever makes is comparative. A team that reaches the fourth step without the second one has bought an expensive change it has no way to evaluate, and will keep it, because it sounds better.
The stall is improving by feel
The stall here arrives after the thing ships. The system answers most questions, the obvious failures are gone, and every change from then on gets judged by trying it: a different chunk size, a rewritten instruction, a newer model, each followed by three questions and a verdict. The activity never slows down, which is why nobody notices that the improving has stopped.
What makes it a stall rather than a habit is who picks the three questions. They are picked by the person who just made the change, minutes after making it, out of the cases that were in mind while making it. Ayşe's second eval set exists because the nephew question was never one of the three and never would have been. Nobody writes a test out of a case they have not thought of yet, which is why the cases have to come from what went wrong rather than from what might.
What to open in 2026
pgvector puts the embeddings in the Postgres already under the application, as of 2026, and it replaced standing up a second database purely to hold vectors. Free and open source. It is the right default because the retrieval that works usually needs a filter beside the similarity search, and a member question about Sunday wants the rules for this wall, currently in force, ranked by both.
Pydantic validates what came back against a declared type before anything downstream sees it, free and open source as of 2026, and it replaced parsing prose with a regular expression and hoping the shape held.
promptfoo runs a file of cases against a prompt on every change and reports which answers moved, as of 2026, free and open source. It replaced trying the same three questions by hand after each edit, which is the method everybody uses first and which quietly selects for the three questions that already work.
All of which leaves the question underneath the month Ayşe lost. Nothing in the retrieval failure was about models: it was a rule written for a noticeboard, indexed by the words it happened to contain. That is getting it, cleaning it, and finding out what it says, arriving in a costume, and the version of this problem that has a name and a literature is getting code off your machine and keeping it running, which had timeouts and retries and rollbacks long before any of this.