What Does 'Done' Mean for AI Systems?
The sign-off meeting with nothing to sign
Last year's portal migration gave the head of operations a familiar ritual: a UAT checklist with 214 test cases, each one green, and a signature line at the bottom. She knew what her name meant on that page. The system did the things on the list.
This year she is being asked to accept a claims triage assistant, and the ritual has quietly disappeared. What she has instead is a demo that went well, a vendor slide claiming "94% on internal benchmarks", and a project that is already late. She signs, because refusing would need an argument she doesn't have the vocabulary for yet.
Six weeks later the assistant misroutes a complaint from a regulator, and the first question from legal is one nobody prepared for: on what basis was this system accepted?
Why the checklist doesn't survive contact with AI
Acceptance testing for conventional software rests on an assumption so basic it is rarely stated: the system's behaviours can be listed. Enumerate them, test each one, and when everything passes, the system is done. Pass is binary and permanent.
A system with a language model in the middle breaks the assumption twice over. The inputs cannot be enumerated, because claims, emails, and contracts arrive in infinite variety. And the outputs are not guaranteed to repeat. "Does it work?" stops being a yes/no question and becomes three harder ones: how often does it work, on which kinds of input, and how badly does it fail when it fails?
The engineering world's answer to those questions is the evaluation suite, or eval. Most enterprises still treat evals as a technical concern, something the data team worries about. That framing is a mistake. An eval is an acceptance test. Which means the business, not the vendor, has to own the bar.
The golden set: requirements, made concrete
The core artefact is unglamorous: a few hundred real examples from your own operation, each paired with the answer your best people agree is correct. Real claims with the correct routing. Real invoices with the correct extraction. Real emails with the correct summary.
Two properties make a golden set worth the effort. It has to be yours: a vendor's benchmark tells you how the system performs on the vendor's idea of your work. And it has to be owned the way a requirements document used to be owned, by the people accountable for the process, with disagreements about "correct" settled before deployment rather than discovered after.
Building one takes your experienced staff a few days of labelling, and this is where many programmes reveal themselves. If nobody can spare two days to define what a right answer looks like, the organisation does not want the system enough to operate it.
Not all errors cost the same
A triage assistant that sends a routine complaint to the wrong queue costs a day of delay. The same assistant closing a fraud indicator costs a regulator conversation. One accuracy number, averaged over both, hides exactly the distinction that matters.
So the bar is never a single figure. Usable acceptance criteria for an AI system look more like an error budget: overall accuracy at or above 92 percent, and severe-category errors, defined in a short taxonomy the business writes, below 0.1 percent with mandatory human review on anything the system flags as uncertain. Site reliability engineers have run on error budgets for years. The idea transfers cleanly; only the vocabulary is new.
Every change is a release
Here is the part that surprises non-technical stakeholders most. A prompt edit feels like tweaking copy. It is a code change, with the same capacity to break things at a distance, and it needs the same treatment: rerun the golden set, compare against the last accepted result, and block the change if a threshold slips.
The same applies to changes you don't initiate. Model vendors deprecate versions on their own schedule, and that schedule will not consult your release calendar. When the underlying model moves, your system has changed whether you touched it or not. A regression suite that ran once, at go-live, protects you from nothing.
Done is a state you maintain
Even a passing system drifts. Your inputs shift when a product line launches or a supplier changes document formats, and performance measured in March says little about November. Mature operations sample a small percentage of production outputs for human review every week, exactly the way call centres have done quality sampling for decades. Nothing about the practice is exotic; what is new is applying it to software.
The clean way to hold all of this together is a split of signatures. The business owner signs the golden set and the thresholds, in advance, when the conversation is calm. Delivery signs the measured result against them. That separation does quiet but real work: "is this bar right for our risk?" is a judgement call, made once. "Did we clear the bar?" is a measurement, repeated forever. Most acceptance arguments in AI projects blow up because both questions are being argued at the same time, after the fact, a week before a deadline.
Which brings us back to the head of operations and the signature line. The point was never the signature. The point, as with accountability in AI governance generally, is that someone can say precisely what they accepted, on what evidence, and what gets watched now that it is live. "Done", for an AI system, means passing and being watched.
Rolling out an AI system with no agreed definition of done? Start with our piece on why pilots fail to graduate, or get in touch to set up an acceptance framework your auditors will recognise.