Why a good final answer is not enough

An AI agent differs from a chatbot not through elegant prose but through its ability to act: read CRM records, create tasks, send email, change statuses and prepare payment documents. A test that asks whether an answer sounds reasonable says little about process safety. The agent may politely report success after selecting the wrong tool, mixing up customers and repeating the operation three times.

OWASP lists excessive agency among the major risks of LLM applications. The root cause is usually not one weak prompt but excessive functionality, excessive permissions or excessive autonomy. Before connecting an agent to production systems, a company must evaluate the whole action path, not only the last sentence.

Three levels of agent evaluation

LangSmith's official documentation separates agent evaluation into final response, individual step and full trajectory. A business needs all three.

1. Individual step

Check whether the agent selected the right tool and produced valid arguments. For “create a reply draft for ticket 481,” the expected behaviour is to read that ticket and create a draft—not export every customer and send immediately.

This test is fast and catches unnecessary tools, wrong identifiers, missing required fields and attempts to leave the permitted scope.

2. Trajectory

Check the sequence of actions. Several valid paths may lead to the same outcome, so a literal list of calls is not always required. Instead, define required and forbidden properties:

  • customer data is read before the reply is drafted;
  • access policy is checked before information is disclosed;
  • a draft exists before approval is requested;
  • nothing is sent externally without approval;
  • one business operation is not executed twice.

Trajectory analysis reveals errors hidden by a good final message. The agent may say “the email is ready” after it has already sent it.

3. Task outcome

Check the business result: the correct record changed, the text is grounded in source data, forbidden changes did not occur and the reviewer received a clear approval request. Deterministic rules and sampled expert review are required; another LLM should not be the only judge.

Architecture of the agent test range

The range should reproduce production interfaces without production consequences. A practical design contains seven parts.

Versioned scenario

Each test stores the user request, initial state of fake systems, available tools, expected trajectory properties and final assertions. The result records model, prompt, tool-schema, policy and fixture versions.

Without versions, evaluation becomes “it seemed to work yesterday.” With versions, the team can identify a regression after changing a model or adding a CRM field.

External-system doubles

Every dangerous integration receives a test adapter with the same contract:

  • CRM returns fictional customers and writes to a separate database;
  • email is delivered to a local mailbox with no internet route;
  • ERP accepts drafts but never posts documents;
  • the payment gateway returns controlled responses and never moves money.

LangChain documentation demonstrates the same principle: replace a real model with a scripted in-memory fake and replace an actual refund with a test function. Tests become fast, repeatable and independent of production credentials.

Action policy gateway

Even in the range, every call should pass through the production-style rules: allowed command list, argument validation, tenant scope, amount limits, mandatory approval and idempotency key. Otherwise, the team tests a free agent and deploys a governed one—or the reverse.

Side-effect ledger

Every action attempt receives a business identifier, parameter hash, timestamp, result and approval reference. The ledger answers concrete questions: how many emails did the agent try to send, did it create duplicates after a timeout, and did it continue after a human rejection?

Tool-action trace

The system needs a technical trace, not a warehouse of secrets. OpenTelemetry warns that tool arguments and results can contain sensitive data. Record operation names, statuses, durations, identifiers and safe hashes by default. Capture content only in an isolated test that uses fictional data.

Assertion suite

Code checks schemas, permissions, call counts, approval ordering and final state. Domain experts evaluate meaning where it matters: reply correctness, summary completeness or escalation quality. An LLM judge can triage results but should not independently approve dangerous behaviour.

Scenarios to include

Start with 30–50 scenarios for one process. Half should represent normal work; the rest should cover boundaries and failures.

  • A normal request with complete data.
  • Two customers with similar names.
  • A missing required field.
  • A user who lacks the necessary permission.
  • An instruction inside an email or document that attempts to override the agent's rules.
  • A timeout after a successful write; the agent must not create a duplicate.
  • Partial failure: CRM updated but email draft failed.
  • Human rejection at the approval step.
  • An attempt to call a forbidden tool.
  • An amount or recipient list beyond the permitted limit.
  • An unexpected response format from an external system.
  • The same request repeated with the same idempotency key.

Include boring cases too: an empty string, a very long name, a wrong time zone and an archived customer. These often turn a smooth demonstration into a late-night log review.

Building realistic test data

Copying a production database into a lab is risky and usually unnecessary. Build a compact synthetic set that preserves process structure: relationships among customers, deals, emails, contracts, statuses and roles.

Important properties include:

  • identifiers look realistic but never match production;
  • ambiguous and conflicting records exist;
  • roles and tenant scopes differ;
  • dates cover overdue, future and boundary conditions;
  • documents contain safe examples of indirect instructions;
  • the expected state after every scenario is known.

Metrics that business teams can use

One success rate hides too much. An agent needs a small metric set:

  • scenarios with the correct final state;
  • wrong-tool selection rate;
  • invalid-argument rate;
  • action attempts without required approval;
  • duplicate side effects;
  • correct refusal rate when data or permissions are missing;
  • recovery after timeout and partial failure;
  • cost and latency per successfully completed scenario;
  • steps per result and unnecessary-call rate.

For dangerous actions, an average is not reassuring. The team needs hard gates, such as zero unapproved sends and zero cross-tenant disclosures across the full suite.

Local deployment does not remove the need for testing

A local model reduces external data transfer but does not repair tool logic. It can still choose the wrong record, retry a command after a timeout or follow instructions embedded in an inbound document.

The test range can run beside the local model server: separate network, fake services, test credential broker and isolated trace storage. The model receives credentials that are technically invalid in production. A single environment-variable change must not turn the sandbox into the real payment system.

Economics of the test range

The main costs are not tokens but scenario design, API doubles and correctness criteria. The suite is then reused after every model and prompt change. The more tools and side effects an agent has, the more expensive it becomes to operate without regression testing.

For a pilot, count:

  • expert hours to create and review scenarios;
  • engineering of safe adapters;
  • compute across repeated runs;
  • errors caught before deployment;
  • time required to accept a new version.

Compare models by cost per scenario that passes every rule, not by token price. A cheap model that calls three unnecessary tools and sends its output to manual review may be an expensive colleague.

A two-week implementation plan

1. Select one agent workflow and list every real side effect.
2. Keep the minimum tool and permission set.
3. Build CRM, email or ERP doubles with matching command schemas.
4. Describe 30–50 scenarios, including failures and policy-bypass attempts.
5. Add deterministic trajectory and final-state assertions.
6. Run each model version several times because behaviour is non-deterministic.
7. Resolve every dangerous failure before shadow deployment on real data.

NIST recommends repeatable and documented test, evaluation, verification and validation processes under conditions similar to deployment. A small business does not need a certification department: it needs a versioned suite, explicit release gates and an accountable owner.

Practical conclusion

An AI agent cannot be accepted from its final message alone. Test tool choice, arguments, trajectory, approvals and the actual state of connected systems. Service doubles let the agent fail cheaply and loudly without touching customers or money.

The first useful artifact is not a universal platform but 30–50 reproducible scenarios for one workflow. If a new version passes them without forbidden actions or duplicates, it can enter shadow mode. Otherwise Vnutrik stays in the lab, enthusiastically sorting cardboard email.