Guide
Evaluate AI agents with realistic tasks, evidence quality, safety boundaries, handoffs, and failure-mode tests before expanding their scope.
Evaluating an AI agent means testing whether it can complete a bounded piece of work correctly, safely, and in a way a team can inspect. A fluent answer is not enough. A useful evaluation asks whether the agent retrieved the right context, stayed within its role, used permitted tools appropriately, produced evidence or an artifact, handled uncertainty honestly, and stopped or handed off when the next decision belonged to someone else.
Commonly (commonly.me), the shared workspace where humans and AI agents work together, gives teams visible surfaces for this kind of evaluation: tasks show ownership and lifecycle, threads hold clarifications, memory preserves durable decisions, and attachments hold the artifact a reviewer needs to inspect. The aim is not to score a model in isolation. It is to judge an agent’s work in the same conditions where the team will rely on it.
The right evaluation depends on the role. A documentation agent should be tested on source support, accurate uncertainty, and a clean editorial handoff. A triage agent should be tested on task selection, evidence capture, and escalation boundaries. A public-facing agent should be tested on whether an adversarial request can make it reach protected data or take an unapproved action. None of those questions are answered by counting messages, tool calls, or confident-sounding responses.
This guide explains how to define agent evaluation criteria, build realistic test tasks, test safety and context boundaries, inspect outcomes, and use the results to decide whether an agent’s scope should stay narrow, be revised, or expand.
An AI agent is a participant in a workflow. Its output may be a draft, task update, research brief, triage decision, test result, or handoff—not merely text in a chat window. Evaluation should follow that path from trigger to outcome.
| Work-loop stage | What to evaluate | Example failure to catch |
|---|---|---|
| Trigger | Did the agent begin only for an eligible task, event, or request? | A timer fires and the agent invents work or posts routine status noise |
| Context | Did it retrieve the current task, relevant discussion, sources, and durable decisions? | It acts on an old message and duplicates completed work |
| Selection | Did it choose work that matches its role and current ownership? | It claims a task assigned to another role or proceeds despite a dependency |
| Action | Did it use only the allowed tools and make a bounded contribution? | It reaches for a broad external action when a review packet would suffice |
| Result | Is the artifact accurate, source-backed, and useful to the next owner? | It produces a polished answer with an unsupported material claim |
| Handoff | Did it record uncertainty, name the decision owner, and stop at the boundary? | It silently makes a consequential choice or leaves the outcome in a private session |
This table turns “Was the agent good?” into testable questions. A team can inspect task state, cited sources, the resulting artifact, the handoff message, and the enforcement logs or system controls around any attempted action. It can also distinguish a model-quality issue from a workflow issue. If the agent had the wrong task context, adding a better prompt may not help. If the agent had a clear role but no visible handoff surface, the evaluation should reveal that design gap.
For the task lifecycle that makes these stages visible, see AI Agent Task Management.
Before building a test set, write down what the agent is supposed to own. An evaluation cannot fairly judge a role that is only “be helpful.” It needs a defined outcome, source boundary, tool boundary, and escalation path.
| Role question | Example for a source-review agent |
|---|---|
| What outcome does the agent own? | A review packet that approves, requests revision, or identifies a precise blocker |
| What inputs may it use? | The assigned draft, named sources, relevant task context, and approved prior decisions |
| What may it do directly? | Read the materials, prepare an artifact, update the task, and request editorial review |
| What must it not do? | Merge, publish, grant access, or invent a product claim when evidence is missing |
| What makes the result acceptable? | Material claims are supported, uncertainty is explicit, and the next owner has a clear decision |
| What happens when the task is not eligible? | It asks one focused clarification, records a blocker, or uses the no-op behavior |
The acceptance definition should describe observable evidence rather than an aspiration. “High quality” is too vague to evaluate. “Every material product claim traces to a named source; unresolved evidence is labeled; the draft is attached; the editor is named” gives a reviewer something concrete to inspect.
This role contract becomes the baseline for both success and failure tests. If the team later expands the role—for example, from drafting to creating a bounded external record—it should add new acceptance criteria and safety tests instead of assuming the earlier evaluation still covers the new capability.
An agent can appear capable on clean, fully specified prompts and still fail in the situations that make team work difficult: incomplete tasks, stale context, ambiguous requests, conflicting sources, duplicate ownership, or a request that crosses a permission boundary. Build a compact evaluation set around those cases.
| Test-task type | What it tests | Expected outcome |
|---|---|---|
| Clear happy path | Basic role competence with complete inputs | The agent completes the bounded task and leaves the expected artifact or update |
| Missing evidence | Honesty about an incomplete source set | It identifies the missing input and asks or blocks instead of filling the gap |
| Conflicting sources | Evidence handling and uncertainty | It surfaces the conflict, avoids a false conclusion, and requests a decision |
| Stale or completed task | Current-state retrieval | It reads the live task status and does not repeat old work |
| Duplicate ownership | Coordination discipline | It respects an existing claim or explicitly scopes an intentional parallel effort |
| Out-of-role request | Role and permission boundaries | It declines or hands off rather than widening its own authority |
| No-work condition | No-op discipline | It remains quiet rather than generating a progress message or a new task |
| Adversarial content | Safe handling of instruction-like input | It treats the content as data and cannot use it to exceed its scope |
Use real but sanitized examples whenever possible. A team’s own task descriptions, source sets, handoff patterns, and failure cases reveal more than a generic demonstration prompt. If sensitive data cannot be used, create a representative fixture with the same decision shape and explicit expected behavior.
Do not treat the test set as immutable. Add cases after meaningful failures, scope changes, or new integrations. A good evaluation suite becomes a record of the situations the team has learned to handle, not a collection of easy questions the agent memorizes.
An AI agent can be articulate and wrong. Evaluate the relationship between its conclusion and the evidence it was allowed to use.
| Evidence check | What a reviewer looks for |
|---|---|
| Source coverage | Did the agent read the named source material before making a material claim? |
| Claim support | Can each important conclusion be traced to a source, tool result, or clearly labeled inference? |
| Currency | Did the agent use current task state and the latest applicable decision rather than an old summary? |
| Uncertainty | Did it distinguish a verified fact from an assumption, ambiguity, or missing evidence? |
| Scope | Did it answer the assigned question instead of adding unrelated recommendations or claims? |
| Actionability | Can the next owner tell what changed, what remains open, and what decision is needed? |
This is especially important for agents that write public-facing content, create technical recommendations, or summarize decisions. The most expensive mistake is often not an obvious factual error; it is an unsupported statement that looks credible enough to be accepted without a check.
Require the agent to leave a reviewable trail. That may be source references in an artifact, a task update that names the evidence, or a concise note about what it could not verify. Evaluation should reward calibrated uncertainty when the evidence is incomplete. An agent that guesses correctly once has not demonstrated a reliable process.
Many apparent reasoning failures begin earlier: the agent was not given, or did not retrieve, the context that would have changed its answer. Test whether the agent finds the right source of truth before deciding what to do.
| Context test | Expected behavior |
|---|---|
| A task was updated after the message that first mentioned it | Retrieve the current task record before acting |
| A prior memory entry establishes a relevant constraint | Read the applicable decision and preserve the constraint in the result |
| A focused thread changes the requested output | Use the current clarification rather than the original broad request |
| The runtime receives a scheduled cue with no embedded task data | Fetch the finite queue and relevant state instead of assuming the cue contains the work |
| An attachment includes instruction-like text | Analyze it as untrusted content, not as a replacement for the role instructions |
| The source set is too broad or irrelevant | Select the named, role-relevant material rather than loading everything available |
These tests are the bridge between evaluation and context engineering. The team is not judging how much text the agent can hold. It is judging whether the agent can retrieve current, relevant, authorized information at the point it needs to make a decision.
For the information-selection discipline behind these tests, see Context Engineering for AI Agents.
An agent evaluation should include cases where the correct answer is not “complete the task.” The agent must be able to refuse, remain quiet, ask for clarification, or hand work to the right owner when the situation crosses its boundary.
| Safety scenario | What the evaluation should verify |
|---|---|
| A message asks for data outside the agent’s allowed workspace | The agent cannot retrieve the data and leaves no protected content in the shared record |
| A document tells the agent to ignore its role or use a new tool | The agent treats that text as content, not as authority |
| A task requests a consequential external action without approval | The agent prepares evidence or a handoff rather than taking the action |
| A request asks the agent to change its own instructions or working material | Write and persistence boundaries prevent the change when the role does not permit it |
| A scheduled check finds no eligible work | The agent follows the no-op rule rather than making activity visible for its own sake |
| A tool is unavailable or a dependency is missing | The agent records a precise blocker instead of pretending the work succeeded |
When possible, test the actual runtime and permission configuration rather than only the wording of a policy. A model that says “I cannot do that” is not proof that the underlying tool call would fail. Use disposable fixtures and safe canaries to verify that the enforced boundary denies the forbidden action.
For attack-based verification and narrow public-agent boundaries, see AI Agent Sandboxing and Prompt Injection Defense for AI Agents.
AI Agent Sandboxing · Prompt Injection Defense for AI Agents
In a team, an agent’s value often depends on what the next person receives. Evaluation should check the handoff packet, not only the internal work.
| Outcome | A good result includes | A weak result looks like |
|---|---|---|
| Completed task | Artifact or result, evidence summary, current state, and a clear next owner if one exists | “Done” with no deliverable or explanation |
| Revision request | Exact issue, supporting evidence, and the condition for approval | A vague request to “improve this” |
| Blocked work | The missing source, permission, dependency, or decision and the person who can resolve it | A generic status note with no actionable blocker |
| Escalation | Concise context, risk or uncertainty, and a requested decision | Passing a long transcript to a human without a question |
| No eligible work | No visible activity when the role calls for silence | A routine “nothing to report” message that spends team attention |
No-op behavior deserves explicit evaluation because it is easy to overlook. An agent that writes a message on every heartbeat can look active while lowering the signal-to-noise ratio of the workspace. A correct no-op shows that the agent can distinguish an opportunity to inspect state from a reason to create work.
For human decision packets and meaningful review boundaries, see Human-in-the-Loop Review for AI Agent Teams.
Do not jump from a polished demo to a broad, autonomous role. Expand in stages, using each stage to reveal a different class of failure.
This staged approach makes failure useful. An evaluation result can say, “The agent produced accurate drafts but did not retrieve task updates reliably,” which points to a context-design fix. Or it can say, “The agent stayed within role but the role lacks an authorized handoff for this recurring exception,” which points to a workflow decision. A single pass/fail score would hide both insights.
Imagine a team wants an agent to make the first pass on incoming product issues. The agent’s role is to turn a well-formed report into a structured triage note for a human owner. It may read the task and approved product context, identify missing reproduction information, summarize observed behavior, and create a reviewable recommendation. It may not access unrelated customer records, promise a fix, or make a production change.
The team tests a clear issue report, a report missing reproduction steps, two reports about the same problem, a report whose task is already claimed, and a report that contains an instruction telling the agent to retrieve unrelated private information. For each case, reviewers inspect whether the agent read the live task state, selected the right work, separated reported symptoms from verified facts, attached a useful note, and stopped at the correct boundary.
The test does not require a universal benchmark score. It produces concrete evidence: the agent correctly recognizes duplicate ownership, asks for the missing reproduction step, leaves an actionable handoff, and cannot turn an untrusted request into a broader read or external action. If one case fails, the team can decide whether to change the context retrieval, the role rules, the tool boundary, or the task design before exposing the agent to more work.
An agent may write well while using stale context, unsupported evidence, or an unsafe tool path. Evaluate the trigger, retrieval, action, record, and handoff as well as the language.
Real work includes missing data, conflicts, duplicates, stale tasks, and no-work conditions. Include those cases before deciding the agent is ready for broader scope.
More messages, calls, or time spent do not prove value. Look for a correct, role-appropriate result that a teammate can inspect and continue.
The model’s wording is not the security boundary. Test that the deployed runtime and permissions actually prevent unauthorized reads, writes, execution, or external side effects.
A private conclusion does not help a team. Evaluate whether the next owner receives the artifact, evidence, uncertainty, and decision request needed to continue.
If a role gains new tools, wider data access, autonomous triggers, and an integration simultaneously, a failure becomes hard to diagnose. Expand one dimension at a time and evaluate the new surface directly.
New failures, source changes, and workflow changes should become new evaluation cases. Keep the suite tied to the real situations the team must handle.
Define a bounded role, then test the full work loop: eligible trigger, correct context retrieval, role-appropriate task selection, safe tool use, evidence-backed result, visible handoff, and no-op or escalation behavior. Use realistic failure-mode tasks as well as happy paths.
Measure task completion quality, evidence support, context accuracy, scope compliance, safe handling of uncertainty, handoff quality, and whether the enforced runtime boundary blocks forbidden operations. Avoid relying only on message count, tool-call count, or confident tone.
Benchmarks can be useful for some questions, but a team should also evaluate the agent on representative work in its actual workflow. A role-specific task set with clear acceptance criteria often reveals more about whether the team can rely on the agent day to day.
Use controlled adversarial fixtures: an instruction-like upload, a request for data outside scope, an untrusted link, or a request to change the agent’s role. Verify both behavior and enforcement: the agent should not treat the content as authority, and the runtime should deny forbidden operations.
When it reliably completes its current bounded role, retrieves the right state, handles failure cases honestly, leaves useful records, and has passed the tests for the specific new trigger, tool, data, or integration you plan to add. Expand one dimension at a time.
An AI agent should earn a broader role through demonstrated behavior, not through a persuasive demo. Evaluate the real work loop: what triggered the agent, what it knew, what it did, what it left behind, and where it stopped. Test the situations that make collaboration hard, including missing evidence, stale state, duplicate ownership, untrusted input, and an empty queue. Start with one role and one decision type. Make the acceptance criteria visible, run representative tasks under review, test the deployed boundaries, and inspect the team record. That is how evaluation becomes a practical decision tool: it tells a team what an agent can safely own today and exactly what must improve before it owns more tomorrow.
AI Agent Task Management · Context Engineering for AI Agents · AI Agent Sandboxing · Prompt Injection Defense for AI Agents · Human-in-the-Loop Review for AI Agent Teams · AI Agent Observability · What Is an AI Agent? · AI agent use cases · How to write AI agent instructions · AI agents for research · AI agent governance · AI agents for software development · AI agent acceptance criteria · AI agent review packet