Choosing and checking business agent environments
A customer asks for a refund because a replacement has not arrived. An agent can give a convincing explanation while refunding the wrong purchase, ignoring an exception or claiming that a payment succeeded after a timeout. To evaluate the agent, we need the customer and order records, delivery events, company policy, permitted tools and a way to inspect the actual consequences independently of its explanation.
A contract-review agent needs a different environment. It must establish which document and amendment govern, interpret their language against the company's requirements, and produce a supported review memo. A correct classification of an isolated clause cannot establish that the full review is complete.
The environment determines what the agent can learn and do, and which failures an evaluation can detect. The business process therefore determines the environment choice. Existing agent code, a large task count or a published leaderboard position cannot make that choice on their own.
What the survey establishes
The supplied catalog describes 81 resources after grouping related versions. It mixes datasets, executable environments, workplace simulations and evaluation tools. Its stated method is a purposeful survey, not an exhaustive registry. It reports source inspection and isolated grader probes, with no complete reproduction of an external environment. Those distinctions remain attached to the resources we select.
The companion source audit identifies exact upstream revisions, README and licence hashes, inspected material and execution limits. It preserves the catalog and archive identities without treating their reported results as independently rerun here. The Support foundation trial and subsequent private agent demonstrations determine the adaptation boundary described below.
Identify which parts of the business process a resource supplies
| Component | What it supplies | What it does not establish by itself |
|---|---|---|
| Dataset | Documents, records or labelled examples | Actions, tool failures, permissions, conversations or useful workflow completion |
| Task collection | Starting requests and intended obligations | A working company system or a complete outcome checker |
| Executable world | State, tools and consequences of actions | Realistic stakeholder knowledge or sound evaluation |
| Actor simulation | Customers, colleagues or events responding during a task | Representative human behaviour or valid business policy |
| Harness | Agent invocation, tool delivery, limits and trace collection | Correct business outcomes |
| Grader | Selected assertions about the attempted task | Coverage of every harmful effect or every acceptable resolution |
| Construction environment | Materials and interactions for building an agent | A production implementation of a particular client's process |
One project may supply several components. The decision record must identify which components are actually reused and which remain our responsibility. Replacing the company policy, introducing a ledger or authoring an amendment is a substantive adaptation that requires its own provenance and checks.
Compare foundations against the intended work
The table records current selection judgments, not measured agent rankings. The tau retail state/tool/evaluator components have now been independently executed as described below. Other external resources remain at source-inspection level in this coordinator audit. No complete external runtime has been reproduced here. Short revisions identify the full pins in the source audit.
| Resource | Strength for the proposed work | Missing work or limitation | Selection judgment |
|---|---|---|---|
| Current tau text domains | Domain policies, customer interactions, tools and task evaluation provide a concrete starting point for post-purchase support. | Native tool effects must be inspected. A return request cannot stand in for a completed financial refund. Our replacement history, authority, consent and durable recovery requirements need explicit mapping. | Adapted selected retail reads and records after reset/evaluator controls. Added authored replacement investigation and custom financial state. No full benchmark reproduction. |
| Hyper-tau | Evaluates constructing an agent from business evidence, inherited systems and stakeholder interaction, then testing the resulting agent. | Its nested construction and serving evaluation involves several model roles and operational dependencies. Running that campaign is separate from using the requirements-discovery method. | Apply its process and knowledge-acquisition ideas. Inspect relevant retail materials; do not launch an automatic construction campaign. |
| ContractNLI | Annotated NDA statements and evidence support document-interpretation component checks. | Fixed hypotheses do not define company exceptions, amendment precedence or a complete review memo. | Adapt evidence components inside a new playbook-based contract environment. Do not market a hypothesis classifier as a complete review workflow. |
| AppWorld | Stateful application APIs and task evaluation support cross-application work. | Our company policy, authority and selected outcomes would still need mapping. Protected app/task assets have additional redistribution requirements described upstream. | Retain as an alternative when cross-application work is essential. Reject as the first primary foundation because the chosen workflows do not yet require its application breadth. Revisit if the Support trial exposes a material gap. |
| The Agent Company | A multi-service workplace supports interactions across business applications and simulated colleagues. | Deployment, service reset and the task-specific business semantics remain necessary. A workplace layout alone does not provide a contract-review playbook or our post-purchase controls. | Retain for tasks whose outcomes depend on colleague and application interactions. Reject for the first cases because those interactions are not required by their current business contracts. |
| AgentDojo | Tool-mediated tasks and adversarial content inform tests of untrusted observations. | Coverage of selected attacks cannot establish complete security, payment correctness or contract-review quality. | Use as a source of adversarial test design alongside the selected business environment. It is not the primary business world. |
Top-level source licences inspected at these pins are MIT for tau, Hyper-tau, The Agent Company and AgentDojo; CC BY 4.0 for the ContractNLI repository; and Apache 2.0 for AppWorld's public portion. AppWorld's README describes a separate protected portion with an added requirement for encrypted public redistribution. Task data, dependencies and exported artifacts still need their own rights checks before reuse or publication. A repository badge does not resolve those questions. AppWorld licence description
The executed Support trial led to a combined adaptation: selected tau reads and records supply the inherited retail system, while the private application supplies exact-money refund, consent and durable recovery controls. Replacement, shipment, contact and exception-policy records are labelled authored additions. This is an explicit component choice, not an adoption of the full benchmark world or grader. The Contract case needs a custom environment around selected document components; installing AppWorld or a workplace simulation would not remove that obligation.
What the first executed Support controls showed
On October 19, an independent coordinator run reproduced the pinned retail component trial on a public development task. The official return operation and an alternative containing an extra lawful read passed its database check. No action and a return combined with an unrelated customer-address change failed. Reset restored the original database identity. The return operation changed the order to a return-request state and left payment history unchanged.
The trial also introduced a backend exception through the real return-tool method. The selected native database grader then awarded success to a no-op, although the independently specified return obligation remained unmet. This is a specific failure of that component under the injected backend fault. It does not establish failure of the complete benchmark or its language-based checks.
The run used a disclosed package-import adapter, the native tool/state/evaluator code and a deterministic public reference action. It did not run a customer simulator, model-selected agent trajectory, language judge or the complete runtime. The result supports keeping an independent business-obligation check and explicit financial semantics. That component trial alone does not establish suitability of the finished adaptation or agent performance. The public source audit retains upstream source identities; private execution records support these checks.
What the working Support adaptation demonstrates
The private frontier agent now investigates a failed replacement through actual tools. In one retained run it finds the original purchase, replacement history, carrier-confirmed loss, applicable policy and previous contact, obtains the configured trusted exception and fresh customer confirmation, then commits one USD 46.00 synthetic refund. Independent checks and the stored SQLite ledger agree on the purchase, exact amount, resolved case and unchanged unrelated orders.
Two additional real-model runs exercise different outcomes. When the customer changes their preference to another replacement, the model opens an evidence-backed human review case and issues no refund. When a committed refund response is deliberately lost, it chooses reconciliation and recovers the original receipt without another refund attempt or duplicate payment. Separate tool/state controls check forged authority, missing proof, revoked consent and a changed exception policy.
These are development demonstrations using authenticated headless model decisions adapted into domain-tool calls. They establish the retained behavior, not unseen accuracy, comparative model advantage, client outcomes or production reliability. The public candidate displays sanitized recordings; the private app remains separate. The recorded Support case identifies the exhibited implementation and its independent checks.
What the working Contract adaptation demonstrates
The second private agent reviews an unchanged public Common Paper NDA together with clearly authored cover choices, amendments, custody records and company playbooks. Its retained full memo distinguishes the supported simulated amendment from a later competing copy, preserves qualified confidentiality and disclosure duties, and identifies both a 180-day versus 90-day retention exception and an unavailable security exhibit.
When the operator changes the pilot from synthetic data to real customer transaction records in ordinary prose, the model records the changed intent, rereads the current context and produces a focused handoff requiring separate Security/privacy and Legal review. A separate real-model run under an explicitly revised 180-day policy changes the retention assessment while retaining its conditions and the missing exhibit. Neither run authorizes signatures, purchases or data access.
Independent review checked the actual memos against original source passages and the selected policies. Source and quotation checks establish evidence identity separately from interpretation. An interrupted first attempt and an unresolved categorical grading disagreement remain recorded. These are development demonstrations within one public source family, not unseen accuracy or legal approval. The public recordings include the actual accepted memo and follow-up.
Gather requirements before defining a successful run
An agent cannot reliably act on company rules that the implementation never acquired. The process dossier records the intended result, current work, actors, systems, permissions, routine cases, exceptions and simpler alternative. It also records source authority, unanswered questions, contradictions and the person responsible for resolving each consequential ambiguity.
The following example is an authored stakeholder exercise for a synthetic company. It is not a client interview, observed company policy or production recommendation.
| Decision stage | Illustrative Support requirement |
|---|---|
| Initial ambiguity | The written return policy covers purchases but says nothing about a replacement that failed to arrive. |
| Stakeholder question | Does a failed replacement extend the refund deadline, and who can authorize the exception? |
| Authored answer | A support manager may authorize a full refund of the original USD purchase when the replacement is confirmed lost. The authorization must identify the purchase and remedy. Unknown delivery status requires investigation. |
| Recorded requirement | Replacement status and trusted exception authority are separate facts. The customer's request does not establish either one. |
| Agent access | The agent can inspect the authenticated customer's orders and fulfillment events, retrieve policy and request a manager decision. It cannot manufacture approval. |
| Tool permission | A refund requires the correct purchase revision, applicable policy or trusted exception, current customer consent and an exact amount. A target change invalidates the old consent. |
| Independent obligation | The final ledger contains one authorized refund for the intended purchase; unrelated records remain unchanged. Trace checks reject unauthorized attempts even if later actions restore the final state. |
| Customization | If the company removes manager discretion, the admissible result changes to a justified handoff. Policy, tool authority and expected outcome must change together. |
For Contract Review, the corresponding ambiguity might concern whether an approved amendment overrides the base NDA's retention clause. The playbook must identify acceptable retention, authorized document versions and the person who decides an exception. Search results can help locate language; they do not grant authority to waive the requirement.
The demonstration should show one such requirement affecting the agent and its evaluation. An invented interview transcript adds no evidence of real customer discovery. A disclosed exercise can show how the method works and what would need to be learned from a real process owner.
Check the world and the grader before comparing agents
A reference attempt establishes that the task can be completed under the available tools and permissions. A valid alternative establishes that the checker does not require one authored tool sequence. An action-requiring no-op establishes that a plausible final explanation cannot substitute for doing the work. These controls must execute against the selected world and the real checker.
| Failure to introduce through the real producing code | Required observation |
|---|---|
| Correct requested action plus an unrelated update or deletion | The checker rejects the extra effect. |
| Refund retry after a committed write whose response was lost | Reconciliation finds the original receipt; no second refund appears. |
| Correct numeric amount with the wrong currency or purchase | The checker rejects the transaction. |
| Customer changes the requested purchase or withdraws consent | The old authorization cannot execute the new remedy. |
| Contract memo omits a required issue or cites an inapplicable version | Coverage or source checks reject the memo even if the prose is plausible. |
| Relevant amendment or policy is unavailable | A qualified unresolved result can be valid; invented certainty cannot. |
| Budget exhaustion before resolution | The run is recorded as unresolved or handed off, with actual effects accounted for. |
The retained independent Agent-Diff and CommerceAgentBench probes found selected grader omissions under reduced fixtures. They justify testing requested effects together with unrelated changes and financial identity. They do not establish full-server exploitability, failure of every task in either benchmark, or a performance estimate for our agents. Their source pins, inputs and execution limits remain in the enterprise-agents evidence record.
The environment must separate authoritative state, the agent's permitted observations, another actor's private facts and evaluator-only expectations. A code-capable agent requires an actual process or container boundary that denies access to hidden files and direct database mutation. Sharing a Python process or changing its working directory does not establish that separation.
After the controls pass, evaluation can compare a simpler workflow, a non-adaptive model workflow and a tool-using agent under a reviewed protocol. Outcomes, grounding, authority violations, handoff, latency and running resources remain separate measures. The agent is not assumed to win. An illustrative successful run demonstrates behavior, not unseen accuracy or production reliability.
Extend to research after the primary cases
Source investigation and computational investigation need different task contracts. A decision brief requires inspected sources, supported claims, conflict handling and explicit unknowns. An experiment requires a fixed protocol, controlled data, actual execution isolation and reproducible outputs, including negative findings.
BrowseComp informs difficult short-answer retrieval; DeepResearch Bench and DeepResearch Bench II inform report and evidence evaluation. None supplies both of our proposed research environments unchanged. These later capabilities do not delay Support or Contract Review. The research source audit also records a discrepancy between DeepResearch Bench's README licence badge and its actual top-level Apache 2.0 licence file; asset-level rights remain to be checked before adaptation.
Show verified work without operating a public agent service
Each public case should connect the disclosed business request to selected requirements, the environment decision, actual model-selected tools, the useful result and independent checks. A failure or unresolved case and a policy customization show where the method's limits and responsibilities sit.
The website receives sanitized recorded artifacts. Visitors may inspect evidence and replay steps, but those controls cannot invoke inference, search, shell commands, business tools or paid jobs. Private operator applications remain separate. Source identity, model and run identity, synthetic material and unexecuted claims must be visible where they affect interpretation.
Source resource SHA-256: d953e6d1c23895801086453003d95afd206c633bdcb5b73b02004c9a94629cb5