The agent is not the product

An agent that drafts a refund email is a writing feature. An agent that issues the refund is an operational system.

That distinction tends to disappear in demos. The model chooses a tool, the tool returns success and the conversation carries on. In production, the interesting questions begin one frame earlier: who authorised the amount, what happens if the payment API times out, and can support tell whether the refund happened?

OpenAI's Agents SDK makes the underlying control explicit: “When a tool call requires approval, the SDK pauses the run”. The useful product decision is where that pause belongs.

Start with consequence, not capability

Imagine an agent helping a support team with a delayed order. It can:

• read the order and delivery history;
• draft a response;
• add an internal note;
• issue account credit;
• cancel and refund the order.

Those actions should not share one permission merely because one model can propose all of them.

A practical policy might allow reads and drafts automatically. Adding a note may require the agent to show its evidence. Credit below a defined threshold might be approved by a support colleague. Cancellation and refund should pause with the exact order, amount, payment method and customer-visible effect on screen.

The boundary is based on consequence, not on how confident the model sounds.

Approval at the start of a task is consent to a plan. Approval at the point of action is control.

Make each tool boring

The safest agent tools resemble narrow internal APIs. They accept structured arguments, validate them again outside the model and return a result the application can verify.

{
  "tool": "issue_account_credit",
  "arguments": {
    "customerId": "cus_…",
    "amountMinor": 1500,
    "currency": "GBP",
    "reasonCode": "DELIVERY_DELAY"
  }
}

This is deliberately less flexible than resolve_customer_problem(instructions). It gives the surrounding product somewhere to enforce currency, thresholds, allowed reason codes and account scope. It also gives an audit log something meaningful to record.

Tool descriptions are not security boundaries. Credentials should be scoped to the operation, arguments should be validated at execution time and the backend should reject states that changed while approval was waiting.

A worked failure case

Suppose the agent proposes a £15 credit. A colleague approves it, the request reaches the billing service and the connection drops before the result returns.

A naive implementation asks the model what to do next. It retries, and the customer receives £30.

A production-shaped workflow does something less theatrical:

1. The application creates an operation ID before execution.
2. The billing tool uses that ID as an idempotency key.
3. A timeout changes the workflow to outcome_unknown, not failed.
4. The application checks the operation status.
5. The agent reports the verified result or asks a person to investigate.

The model can explain the state. It should not invent the state.

Evaluate the whole trace

Single-response scoring misses the failures that matter here. Keep representative workflow traces and assert that the system:

• selected a permitted tool;
• used evidence from the correct customer and order;
• paused before the consequential action;
• preserved the approved arguments;
• handled rejection without repeatedly asking;
• resumed safely after interruption;
• verified the external result before claiming success.

Include hostile and awkward cases: an instruction hidden in retrieved text, a user asking the agent to exceed their own permissions, a price changing after approval, and the same request arriving twice.

The NIST AI Risk Management Framework is useful here because it treats governance, measurement and management as ongoing work rather than a launch checklist.

Autonomy is not the outcome

People do not buy an agent because it can make more decisions. They buy a faster, clearer or less repetitive outcome. Sometimes the best product lets the model prepare everything and leaves one meaningful decision with a person.

That is the approach behind our AI product engineering: expose the uncertain parts, make permissions visible and design recovery before increasing autonomy. A narrow agent that completes one valuable workflow safely is a product. A general agent with a long tool list is still a demo.

Sources

• Human-in-the-loop — OpenAI Agents SDK
• AI Risk Management Framework — NIST
• Making retries safe with idempotent APIs — Amazon Builders' Library