Your AI agent is online. Is the work getting done?

An agent can call its tools successfully and still miss the user’s goal. AWS makes that distinction explicitly in its production-monitoring walkthrough: healthy execution is not proof of useful work. Its examples include a booking agent that stops completing reservations while logs show successful tool executions. AWS

For an existing agent, we recommend a completion loop: define the assignment, check the downstream record, route discrepancies to an owner, and verify the result after recovery. Count the agreed outcome, not the agent’s declaration. Anthropic makes the underlying distinction explicit: a booking confirmation in a conversation is different from a reservation in the database. Anthropic

Decide what completion means

Consider our proposed operating design for an existing order-intake agent. Give it a bounded assignment: create a draft order matching an authorised request, or hand an ambiguous request to a named reviewer. Keep order approval and fulfilment outside that assignment. This applies the evaluation principle of defining inputs and success criteria before judging performance. Anthropic

Before connecting monitoring, we would agree the acceptance evidence with your process owner: request identifier, customer reference, items and quantities, expected draft status, and reviewer destination. These are requirements to investigate, not claims that your ERP already exposes the necessary records.

Keep separate fields for agent assignment and wider business task. Under this contract, an acknowledged escalation can complete the agent’s assignment while the order remains unresolved. We recommend requiring acknowledgement by the receiving owner, rather than treating a sent notification as a completed handoff. Clear roles and a working response record follow established incident-management practice. Google SRE

Run one assignment through checked closure

The following is our recommended workflow, applying Anthropic’s environment-based outcome checks and Google SRE’s explicit incident ownership. It is an implementation proposal, not a description of either company’s order-processing system. Anthropic · Google SRE

1. Record the assignment. Retain the authorised request, expected outcome, acceptance checks and responsible reviewer. Specify whether success means a verified draft or an acknowledged escalation. Do not include approval or fulfilment unless they are explicitly part of the assignment.
2. Read the downstream evidence. When the agent reports a draft created, independently retrieve the order using the agreed identifier. Compare customer, items, quantities and status with the request. Before enabling this check, verify read permissions, identifier matching, record freshness and duplicate detection.
3. Route discrepancies without guessing. If evidence is missing, stale or unreadable, mark the assignment unverified. Do not treat missing evidence as permission to retry. Send discrepancies to the named reviewer and record acknowledgement, missing information and the next action. An unacknowledged handoff remains open.
4. Approve recovery and check again. Require human approval before consequential changes to orders, customer commitments, permissions or agent behaviour. Retain the approved correction and independently check the resulting record before closure. For related failures, name an incident owner and keep an investigation record.

Agree review frequency and escalation deadlines with your business owner before starting. For this design, a matching draft means “draft preparation verified,” not “order fulfilled.” A corrected draft becomes “verified after correction,” not an unqualified first-pass success. These are task-specific acceptance rules, not universal thresholds or measured results.

Keep quality scores beside record checks

The strongest counterpoint is that agent evaluation already addresses completion. AWS’s walkthrough samples a configurable share of production requests and uses model-based judges to assess helpfulness, correctness and goal completion, with reasoning attached to scores. This is not monitoring that ignores outcomes. AWS

Keep those scores. We recommend displaying them separately from record reconciliation: “the evaluator judged this interaction successful” and “the requested draft passed its business-record checks” answer different questions. Use disagreement to select work for review, not to discard either signal. This recommendation draws on AWS’s interaction evaluations and Anthropic’s definition of an outcome as the environment’s final state. AWS · Anthropic

The trade-off in our proposed approach is additional integration and review work. We do not yet know whether your systems expose sufficiently reliable evidence. Verify that access before promising automated closure; where it is unavailable, keep the result unverified and assign a person to resolve it.

Make the first decision small

We would start by mapping one agent assignment to its authoritative business record, implementing the acceptance check, and operating the discrepancy queue with your process owner. Keep corrected examples for subsequent evaluations; Anthropic describes how evaluations make failures and behavioural changes visible across an agent’s lifecycle. Anthropic

Your first decision is whether one representative assignment can be closed with evidence. Test a matching draft, an unreadable record and an unacknowledged handoff. Expand only after your team can demonstrate what closes the assignment—and who owns the work when it does not.