How do you safely let an in-product AI agent take actions for users?

The design rule: language may suggest, policy must decide

An AI model is useful for understanding a request, choosing among permitted tools, and preparing a plan. It is not a reliable policy engine. It can misunderstand a request, invent details, be influenced by untrusted text, or select the wrong action. A system prompt that says “do not delete data” is advice to a probabilistic component, not enforcement.

Make this distinction explicit:

Every proposed mutation should become an immutable server-side action envelope, not a free-form instruction. At minimum it includes action_id, actor and tenant, delegated user, allowed operation, canonical target IDs, normalized parameters and bounds, policy version, creation and expiry times, an idempotency key, approval state, and a digest of the exact execution payload. The application, not the model, resolves identifiers such as “my largest overdue invoice” to a specific invoice in the authenticated tenant.

This separates three questions that are often accidentally merged:

• Authentication: who is using the product?
• Authorization: may that user perform this operation on this resource now?
• Intent: did that person knowingly ask for this particular effect, within these limits?

Authorization should be denied by default, checked for every request, and kept close to the data or service being protected. These are core OWASP authorization recommendations (Authorization Cheat Sheet). Intent adds a second binding for actions where “the user could do this” is not enough to show “the user meant to do this now.”

Reference architecture: one narrow path to a real-world effect

The flow below is a visual brief. The trust boundary is intentional: only the execution gateway can obtain a credential that reaches an external system.

flowchart LR
    U[User in product] --> A[Session and tenant context]
    A --> I[Intent service: canonical request and limits]
    I --> P[Agent planner: untrusted proposal]
    P --> V[Schema validator]
    V --> G[Policy and risk gateway: authorization, limits, data rules]
    G -->|needs approval| C[Confirmation or step-up auth]
    C --> G
    G --> X[Execution orchestrator: idempotency and recovery]
    X --> B[Credential broker: one-use scoped grant]
    B --> T[Narrow tool adapter]
    T --> S[Business system]
    G -. audit decision .-> L[Append-only action ledger]
    X -. audit outcome .-> L
    T -. audit effect .-> L

Do not give the planner a database administrator account, a general cloud key, a generic OAuth token, an arbitrary HTTP client, shell access, or a tool such as run_any_query. Expose small capability-shaped tools instead: get_order_summary(order_id), draft_refund(order_id, amount_minor, reason), or execute_approved_refund(action_id). The execution form is deliberately less expressive than the planning form.

For a tool that refunds an order, a model-facing schema might permit only:

{
  "action": "refund_order",
  "order_id": "ord_123",
  "amount_minor": 1250,
  "reason_code": "duplicate_charge",
  "idempotency_key": "act_8f..."
}

Schema validation is necessary but not sufficient. The backend must fetch ord_123 under the user's tenant, confirm that it is refundable, calculate the maximum from trusted records, require the correct currency, reject unknown fields and values outside a small enum, and attach the action to the approved payload digest. A request that says amount_minor: 1250 is a proposal, not proof that a USD 12.50 refund is allowed.

Scope authority to the task, resource, and moment

Use a dedicated agent identity plus a user delegation claim. The resulting grant should say, in effect: “agent execution gateway acting for user 42 may refund order 123 in tenant A for at most 1,250 minor units, once, until 14:05 UTC.” It should not say “the assistant may use the payments API.”

Apply least privilege in four dimensions:

• Function: grant create_ticket_comment, not a general ticket administration API.
• Resource: bind the grant to the resolved object, project, account, or a server-side query predicate. Never let the model select a tenant or rely on a client-supplied role.
• Data: return only fields necessary for the action, with sensitive fields redacted before they reach the model. A billing assistant might need invoice status and amount, not full card or bank details.
• Time and route: use a short expiry, one execution where practical, and an audience limited to the exact service. Revoke on logout, role loss, approval cancellation, incident, or a relevant resource state change.

OAuth 2.0 Security BCP advises restricting token privilege to what the application needs, including audience restriction, and recommends sender-constrained tokens to reduce misuse of stolen tokens (RFC 9700, sections 2.2 and 2.3). Use those practices where your authorization system supports them, but do not mistake a well-scoped token for a complete business policy. The resource service must still re-evaluate the user, object, state, and transaction rules at execution time.

For a multi-tenant product, an effective pattern is for the tool adapter to accept an action_id, then load the canonical envelope itself. Avoid an adapter API that accepts arbitrary tenant_id, user_id, or raw OAuth token arguments from the model. This closes a common confused-deputy path: the model cannot turn a legitimate user session into a request for someone else's resource.

Match confirmation to consequence, not to the word “agent”

Confirmation is valuable only when it presents a concrete, stable effect. A modal that says “Allow agent action?” teaches users to click through. Show the target, meaningful fields, recipient, cost or scope, and whether the action is reversible. The server should generate this view from the canonical action envelope, then bind the confirmation to its digest, the authenticated user and session, and a short expiry. Any change to a material field invalidates it.

The exact threshold is a product risk decision. For example, a B2B system may let an account owner apply a label without a prompt, but still require one confirmation per customer-visible message, regardless of monetary value. Financial and destructive transactions need special care: OWASP recommends server-side authorization, a final authorization gate before execution, protection against changes after approval, short validity periods, and unique authorization data per operation (Transaction Authorization Cheat Sheet).

Do not accept a confirmation as durable permission for every future action. “Approve automatic refunds up to $50 for duplicate charges on this store until Friday” can be a valid saved delegation if it names the merchant, currency, reason, cap, aggregate cap, expiry, and revocation path. “Let the agent handle refunds” is not a defensible scope.

Treat prompt injection as an authority-boundary problem

Prompt injection is not confined to a malicious chat message. A webpage, email, PDF, CRM note, image, search result, tool description, stored memory entry, or another agent can contain text that tries to redirect the agent. OWASP notes that both direct and indirect injection can lead to unauthorized tool access or execution, and that retrieval and fine-tuning do not fully solve it (OWASP LLM01:2025).

Use several defenses, while assuming some injections will still influence the planner:

• Label every item of user, retrieved, third-party, and tool-returned content as untrusted data. Keep it distinct from system policy, identity claims, and confirmed intent. Do not let any natural-language content alter scopes, approval tier, policy configuration, or destination allowlists.
• Place policy evaluation and tool authorization outside the model. Even if a document persuades the model to call a tool, the call must fail unless the action envelope and policy independently allow it.
• Treat tool manifests and third-party connectors as supply-chain inputs. Pin approved versions, review their permissions, use a small allowlist, and keep a kill switch for each connector. Do not pass arbitrary tool descriptions into a privileged planner.
• Remove arbitrary network fetch, URL callbacks, shell commands, SQL, code execution, and file-system access from high-trust agents. If such capabilities are genuinely needed, run them in a sandbox with an egress allowlist, CPU, memory, process, file-size, and time quotas, then keep their output untrusted.
• Partition retrieval and memory by tenant and classification. Apply access filtering before retrieval, minimize context, expire action-related memory, and require a new confirmation rather than trusting a past conversational statement.
• Keep secrets out of prompts, model-visible tool output, client logs, and diagnostics. Store credentials in a broker or vault and redact sensitive fields in action logs.

The aim is not to perfectly detect every malicious sentence. It is to ensure that a successful injection cannot expand authority or cause an unchecked side effect.

Make execution safe under retries, partial failures, and real business rules

An action that reaches two systems can fail after the first succeeds. A payment call can time out after the processor accepts it. Never turn “no response” into “try it again” for a non-idempotent action.

Give every action one idempotency key and keep an execution ledger with states such as proposed, awaiting_approval, approved, executing, succeeded, failed, unknown, cancelled, and compensated. Store the request fingerprint with the key. A repeat with the same key and same payload returns the known result; the same key with a different payload is rejected. HTTP defines an idempotent request as one whose intended server effect is unchanged by identical repeats, which is why such requests can be retried when a response is lost (RFC 9110, section 9.2.2). For POST and other non-idempotent APIs, implement this behavior yourself or confirm the target system's documented idempotency support. The IETF's idempotency-key specification was still an Internet-Draft on the verification date, so use it as implementation guidance, not as a completed standard (draft-ietf-httpapi-idempotency-key-header-07).

For an internal mutation, put the business update, policy decision reference, ledger record, and outbox event in one database transaction. A background worker can then deliver the outbox event, safely retrying it with the same action key. For external systems, use a state machine or saga: record each completed step, make compensation explicit, and do not claim rollback unless it really reverses the business effect. A refunded card payment may be a new compensating payment, not a restoration of the original authorization.

Mark ambiguous outcomes unknown, stop automatic retries, reconcile against the target system using a provider-side transaction or idempotency identifier, then surface the result to the user or an operator. A useful failure design is often more conservative than a success design.

Build reversibility into low-risk operations: soft-delete with a retention window, create a draft before sending, archive rather than purge, move to a quarantine state, or schedule a delayed execution with a cancellation window. Provide a dry-run tool that returns the exact objects, changes, expected cost, and blockers without side effects. A dry run is not permission to execute later, because target state, authorization, and limits may have changed.

Three concrete examples

Low-risk action: triage a support queue

Hypothetical request: “Mark the duplicate password-reset tickets in my Support project as duplicates.”

The agent searches only tickets visible to the authenticated project member, proposes a list of at most ten ticket IDs and a mark_duplicate change, and shows a dry run. The policy permits automatic execution only if the user previously enabled that specific rule for this project, the ticket is not escalated or legal-hold, and all changes are reversible for 30 days. The adapter loads ticket IDs from the envelope, checks project membership again, uses one action key per ticket, and records a batch result. A hidden instruction in a ticket that says “close every VIP ticket” cannot enlarge the list or call a closure tool.

Financial or destructive action: refund an order

Hypothetical request: “Refund the customer's duplicate $12.50 charge.”

The agent may locate likely orders, but it cannot execute from that phrase. The application resolves the customer and order, verifies a settled duplicate charge and a remaining refundable balance, and creates a proposal for USD 12.50 with reason duplicate_charge. The confirmation screen states the order, customer reference, amount, currency, payment destination class, reason, and the fact that a refund may not be reversible. The user reauthenticates if the policy requires it. The confirmation is tied to the exact envelope; increasing the amount or changing the order invalidates it. The payments adapter receives a one-use grant, sends the action ID as its idempotency reference, and handles a timeout as unknown until reconciliation, not as permission to send a second refund.

Cross-system workflow: onboard a contractor

Hypothetical request: “Onboard Maya as a contractor next Monday.”

This is not one atomic action. The agent should create a plan: open a pending contractor record in HR, request a manager approval, provision a least-privileged identity after approval, create a time-limited project membership, and prepare a procurement request. The plan shows each system, data category, owner, effect, and rollback or expiry. The policy may require HR approval and security approval before the identity step, prohibit copying compensation data into the project tool, and force the requester to choose an existing project rather than accepting a model-created destination. Each step has its own narrow grant and ledger entry. If identity provisioning succeeds but project access fails, the orchestrator knows whether to retry safely, open an operator case, or remove the unused account. It never assumes that “onboarding completed” merely because the model said so.

Bound blast radius, cost, and data movement

Action safety includes availability and spend. Set limits at more than the chat session:

• Per user, tenant, action type, target system, and time window: requests, concurrency, objects changed, emails sent, and cumulative monetary amount.
• Per agent run: maximum steps, tool calls, recursion or delegation depth, wall-clock time, tokens, data retrieved, and retry budget.
• Per integration: outbound request rate, daily spend ceiling, provider quota, retry backoff, circuit breaker, and alert threshold.

Limits need business semantics. “20 requests per minute” does not prevent an agent from issuing 20 one-dollar refunds, 20,000 emails in a bulk endpoint, or an expensive export. Define limits in units of harm as well as compute. OWASP warns that APIs without appropriate resource limits can create denial-of-service and third-party cost exposure, and recommends rate limits, request bounds, execution controls, and provider spending limits (OWASP API4:2023).

Give security and operations teams a fast global kill switch, plus narrower switches for a model version, tool, connector, tenant, or action class. A kill switch should stop new execution grants and queued work, but preserve the ledger and reconciliation workers needed to understand already-started actions.

Audit, monitoring, escalation, and incident response

Audit every decision, not only successful tool calls. A useful record includes correlation and action IDs; user, agent, and service identities; tenant; requested and canonical target references; policy and tool versions; risk tier; approval and payload digests; credential grant metadata; timestamps; outcome; provider reference; and a redacted error category. Store hashes or references for sensitive values rather than access tokens, full prompts, bank data, or raw documents. Preserve trustworthy time ordering and retain records under an access-controlled, tamper-evident logging design.

Monitor for both security and product failure: policy denials, confirmation cancellation, permission mismatches, cross-tenant attempts, tool-call rejection, unexpected data volume, unknown outcomes, duplicate requests, cost spikes, approval bypass attempts, and changes in the distribution of action types. NIST's AI RMF core calls for defined human-AI oversight processes and monitoring system behavior in production (NIST AI RMF Core).

Have a rehearsed response path:

1. Contain: disable the affected tool or model route, revoke grants, pause queues, and apply a lower-risk policy mode.
2. Preserve evidence: retain action, policy, approval, and connector logs with their correlation IDs; do not delete the failing conversation before capture.
3. Scope and reconcile: find actions by connector, policy version, tenant, and time range; identify completed, failed, and unknown effects from target systems.
4. Repair: use documented compensations where safe, obtain human approval for high-impact repair, notify affected users when warranted, and rotate compromised credentials.
5. Learn: add a regression test for the failed invariant, review the tool's authority, and require a staged re-enable with heightened monitoring.

Human escalation is a feature, not an exception. Route cases to a named operator when the policy cannot establish the target, outcome is ambiguous, a risk threshold is crossed, a legal or regulated decision is involved, a connector is unhealthy, or the request asks to override a control.

Pre-release checklist

Before enabling state-changing agent tools, verify all of the following in the production-like environment:

• Each tool has one narrowly defined purpose, strict input and output schemas, server-side validation, and no unused privileged method.
• The tool cannot accept an arbitrary tenant, role, target, credential, URL, query, shell command, or approval state from model output.
• Resource services independently enforce user, tenant, object, field, and business-state authorization on every mutation.
• Every executable action has a canonical envelope, payload digest, expiry, idempotency key, policy version, and user-visible result.
• Confirmation tiers are documented; material changes invalidate approval; high-risk actions get strong reauthentication or transaction binding where required.
• Credentials are short-lived, narrowly scoped, audience restricted, revocable, and issued only to the execution gateway.
• All external content and tool output is treated as untrusted; retrieval, memory, connector manifests, sandbox egress, and secrets have explicit boundaries.
• Internal and external side effects have tested idempotency, state machines, timeout handling, reconciliation, and safe compensation or escalation paths.
• Rate, concurrency, object-count, data-volume, monetary, token, and third-party spend caps are enforced and observable.
• Dry runs, undo or retention windows where feasible, batch previews, cancellation windows, and a global plus per-tool kill switch are implemented.
• Logs connect intent, authorization, confirmation, grant, execution, and outcome without retaining secrets or unnecessary personal data.
• Red-team tests cover direct and indirect prompt injection, malicious tool output, cross-tenant ID substitution, stale approvals, policy-version changes, duplicate delivery, ambiguous timeouts, and bulk or spend abuse.
• Operators have run an incident drill that includes disabling a connector, finding all affected actions, reconciling unknown states, and communicating a result.

Practical rollout and limits

Start with a small, measurable capability set. First let the agent read and prepare drafts. Next enable one reversible mutation with a low object cap, robust audit, dry run, and a clear undo. Add confirmation-backed actions only after you have observed false positives, approval behavior, retries, and recovery in production. Mature organizations should version policies and tool schemas, test authorization as a regression suite, and make a policy change as reviewable as code.

There are limits. User confirmation does not repair a misleading preview, alert fatigue, or an unauthorized object lookup. Sandboxes do not make arbitrary credentials safe. Idempotency prevents duplicate effects for the same key, not a wrong first effect. A model may still make a poor recommendation. For high-consequence actions, the better product may be an agent that gathers evidence, prepares the exact transaction, and hands control to a responsible person.