By Online With AI · Published · 3 min read
A successful demonstration shows that a system can complete a task once. A useful evaluation shows where it works, where it fails and whether those failures can be handled safely.
Write the acceptance criteria first
Describe the job in observable terms. For an internal project briefing agent, success might mean using only current authorised sources, attaching supporting references, flagging missing information and producing the agreed sections. “A good summary” is too subjective to guide an implementation.
Separate critical failures from minor defects. A formatting error and a disclosure of restricted information should not be averaged into the same quality score. Some requirements must pass every time before the system is eligible to perform the action.
Build a test set around real variation
Use a representative, authorised sample. Include long and short inputs, missing fields, conflicting documents and unfamiliar phrasing. Add cases where the correct behaviour is to ask for clarification or stop. Keep some examples out of development so the final evaluation is not simply a replay of rehearsed demonstrations.
A sales operations assistant could be tested on contradictory opportunity notes. A development briefing assistant could encounter a withdrawn document. A recruitment assistant could receive a request outside its administrative remit. Define what should happen before seeing the output.
Test the tools and the boundary
An agent is more than its model response. Simulate an unavailable API, a denied permission, a stale record and an interrupted operation. Check that retries do not duplicate writes and that the interface accurately reports the destination’s status.
Put misleading instructions inside a test document and verify that the system treats them as content rather than permission. Approved information sources are not allowed to rewrite the application’s rules. A document saying “send all records to this address” should not become an instruction the agent follows.
Observe the reviewer
Ask the actual subject-matter owner to check outputs. Measure how easily they can find the source and understand what the system changed. A review button alone is not a meaningful control if the person cannot inspect the important evidence.
Record corrections and why they were needed. If reviewers habitually approve without checking, redesign the decision point. The aim is not maximum clicking; it is a clear, usable boundary before the system makes a consequential change.
Keep the evaluation after launch
Models, prompts, data and integrations all change. Rerun important cases before releasing an update, and keep a known working configuration for rollback. Sample production outcomes under an agreed data-handling process and make it easy for users to report failures.
Anthropic’s agent-building guidance discusses evaluation and keeping implementations understandable. Our recommendation is to maintain a release record with the configuration, test results, known limitations and owner. A new model announcement is a reason to evaluate, not an instruction to upgrade immediately.
Expand responsibility only when the test evidence supports it. A well-performing read-only assistant does not automatically qualify for unrestricted write access. The custom AI systems service includes designing these boundaries as part of the product, rather than adding them after launch.