A complete, polite answer from a model makes a demo end nicely. But an agent that operates business systems can also look up the wrong record, miss a key condition, or claim to have done something it never actually did. Acceptance has to follow the business result.
Check the wording, the process and the result separately
Anthropic’s article on agent evaluation distinguishes the transcript of what ran from the final outcome in the environment: a system saying a task is done does not prove the business record exists.[1] For business delivery, it’s a reminder to separate the levels of acceptance.
At the wording level, check the answer is clear; at the process level, check which information and tools were used; at the result level, verify the state is as expected. A support lookup should use the right order, ask for more when information is missing, and after creating a ticket return a record that really exists. None of the three can stand in for another.
Build a representative set of tasks first
Examples can come from authorized, anonymized past questions, or from clearly labeled simulated scenarios. They should cover everyday tasks, missing information, ambiguous records, conflicting rules and service failures. We suggest keeping the first set small, but making the expected result of every task unambiguous.
“A customer asks about delivery time”, for instance, needs the order status, the information allowed and the range of acceptable answers spelled out. If the API has no estimated arrival time, a passing answer admits that; it must not make up a date. A good test case is usually also a clear requirement.
Don’t squeeze all quality into one score
Accuracy, human takeovers, response time and cost per task each reveal different problems. A change may make answers more complete but add a lot of waiting; or it may cut takeovers while letting exceptions run that should never be automated. Rolling them into one total score hides important trade-offs.
Set the conditions each type of task must meet, then watch the experience metrics. Overstepping permissions, acting on the wrong record and unconfirmed writes should each count as a failure on their own. For whether replies are concise and readable, use an explicit rubric, and have business staff recalibrate the scoring regularly.
Evaluate with every change
After changing the model, prompts, retrieval rules or APIs, compare against the same set of tasks. Where outputs vary, run the key tasks several times and record the versions and settings, so a lucky success isn’t mistaken for a lasting improvement. Automated scoring helps widen coverage, but contested business judgments still need a human review.
Failures after launch should go into a queue for analysis. Once anonymized and confirmed, add representative failures to the test set so the same problem doesn’t keep coming back. Also keep some tasks out of day-to-day tuning, to reduce the risk of a system that only does well on examples it knows. Evaluation then becomes a shared language for requirements, development and operations.
At your next acceptance check, pick an action the system says is “done”, and go into the target system to verify the result exists and belongs to the right record.
References
This article combines public sources with SWYZL & Co.’s recommended methods. The hypothetical scenarios illustrate design thinking and do not represent client results.
