How should AI agents be prevented from taking damaging production actions without approval?

The control model

The useful mental model is a four-part system: prevention stops an unsafe action before it starts; detection notices an action or pattern that escaped prevention; containment stops the blast radius; recovery restores the intended state and learns from the failure. Approval is one prevention control. It is not the whole system.

An agent should be able to ask for a capability such as refund_order, send_campaign, delete_customer_export, or deploy_service_version. It should not hold a general administrator token, raw database connection string, generic shell access, or a tool called run_any_command. A capability needs a narrow schema and a policy contract: which tenant or project it can touch, permitted fields, maximum item count, maximum value, allowable recipients or destinations, environment, expiry, and whether an approval is required.

The application, not the language model, should turn that proposed capability call into a production request. A trusted execution gateway then authorizes it just before the side effect. That gateway is the right place to apply identity and access management, business constraints, cost limits, approval verification, transaction handling, logging, and circuit breakers. NIST describes least privilege as allowing only access necessary for assigned tasks, including for processes acting on behalf of users, and recommends logging execution of privileged functions. NIST SP 800-53 AC-6

Authorization at execution time

Authorization at execution time means the gateway evaluates the request against the facts that are true when it is about to run, not merely when the model planned it or when someone opened an approval screen. At a minimum, check the authenticated requesting user or service, the agent identity and version, tenant, action type, target identifiers, resource ownership, field-level scope, environment, current record version, active policy, available budget, rate limits, and an unexpired approval if required.

This recheck matters because context goes stale. A customer may resolve a ticket after the agent read it, an order may already have shipped, an account's permissions may have changed, or a new incident may make deployments unsafe. For a write, use an optimistic concurrency check or equivalent version condition so that a preview based on version 42 cannot overwrite version 43. If the condition fails, return a conflict and make the agent build a new proposal. Do not silently apply the old plan to the new state.

Give production actions their own short-lived service identity. Prefer a workload identity or narrowly minted token over a long-lived secret embedded in an agent process. The token should encode or reference only the approved capability and scope, and the destination service should verify it. Keep the model process unable to bypass the gateway by directly reaching a database, payments provider, mail API, cloud control plane, or production credential store.

A risk-tiered action policy

The exact thresholds are a business decision, but the distinction between action classes should be explicit and testable. The following starting point is deliberately conservative. An action moves up a tier when it affects more records, crosses a tenant or trust boundary, sends outside the organization, changes permissions or infrastructure, creates financial liability, handles regulated data, or becomes hard to reverse.

The point of the table is not to label an operation once. A small refund can be medium impact for a single order and high impact as a batch or when repeated. Similarly, a message is not low risk merely because the send API is simple. Recipient count, domain, content category, and whether the message is public or contractual can change its tier. Make the policy engine calculate these factors from trusted request data rather than accepting the model's label.

Prevention controls

Use scoped capabilities rather than broad tools

Split tools by action and authority. For example, expose get_order_summary, propose_refund, and execute_approved_refund instead of a generic commerce API client. A record-editing tool should accept only the allowed fields and identifiers, not arbitrary SQL, arbitrary JSON patches, or free-form URLs. A sending tool should enforce recipient ownership, approved templates where appropriate, attachment rules, and a maximum batch size. A deletion tool should usually create a reversible deletion request, not issue a hard delete.

This design reduces both accidental mistakes and prompt-injection damage. Untrusted text in a ticket, web page, attachment, or email may try to persuade the model to call a tool. It cannot enlarge the tool's independently enforced target scope, permission, or limit. OWASP's current agent guidance emphasizes inspectable agents and runtime policy enforcement rather than relying on a model to police its own authority. OWASP Agent Control Standard

Build separate identities for development, staging, and production. A staging agent should have no credential that production services recognize. If an agent needs code execution, use an isolated workspace with minimal mounted data, restricted network egress, resource quotas, and no production administration path. Sandbox testing validates behavior, but it does not replace production authorization because the safety boundary must still work after a new model, tool, or workflow is deployed.

Make the proposed effect inspectable

For a consequential action, make the first tool call a plan or dry run. The trusted service, not the model, should return a receipt containing the exact action, targets, current versions, before and after differences, downstream systems, price or count, policy result, known irreversible effects, and a proposed idempotency key. The model may present that receipt to a reviewer, but it must not be able to edit the receipt or substitute target IDs before execution.

An approval must bind cryptographically or by server-side reference to that receipt. Store an execution ID, receipt digest, approver identity, decision time, expiration, permitted quantity and amount, and policy version. On execution, the gateway compares the requested action with the approved receipt and rejects any mismatch. An approval for refunding order A123 for $25 is not approval to refund A124, refund $250, add a recipient, or repeat the call next week.

Present approvals in a reviewer interface designed for a human decision, not a model-generated narrative. Show the real target names and IDs, impact count, money, before and after data, external destinations, reversibility, and reasons the policy selected the tier. Require step-up authentication for sensitive decisions and route them to a person authorized for that business domain. Avoid an approval button whose only evidence is the agent's prose.

OpenAI's Agents SDK demonstrates the appropriate control-flow property for tool approval: when a tool call needs approval, it pauses before execution and resumes only after an explicit approve or reject decision. That is an implementation feature, not a complete policy system, but the pause-before-side-effect behavior is the one to preserve in any stack. OpenAI Agents SDK human-in-the-loop guide

Apply budgets, rate limits, and transaction discipline

Enforce spend limits and rate limits at the gateway and, where possible, at the downstream provider. Set ceilings per action, user, tenant, agent, time window, and workflow. Examples include a maximum of one refund per order, $100 per agent per day, 20 external messages per minute, and no more than 50 records in a single approved batch. The values are illustrative. They should come from product, finance, security, and operations owners. A model instruction such as "do not spend more than $100" provides no enforceable protection if the payment credential itself has no limit.

Every mutating request should carry an idempotency key derived from the approved execution ID. If a timeout causes a retry, the downstream service must return the result of the original accepted request rather than repeat the charge, email, deployment, or delete. Retain deduplication records for a period that covers realistic retries and queued work. Treat a missing idempotency guarantee as a reason to lower the action tier or add a wrapper that supplies one.

Use a database transaction when all intended changes are inside one transactional system. The transaction should validate constraints and write the authoritative audit event before commit, or use a transactional outbox so downstream events are emitted reliably after commit. For effects that span a database, payment provider, email system, and cloud provider, do not claim atomicity that does not exist. Use a saga or workflow with explicit state, bounded retries, compensating actions where possible, and a reconciliation queue for partial failures.

Design for reversibility and safe failure

Prefer soft delete, versioned objects, staged configuration changes, feature flags, and an undo period over irreversible deletes and immediate public sends. A reverse operation must itself be authorized and logged, because an attacker or malfunctioning agent could otherwise use "undo" to create a second harmful change. Some effects cannot be truly reversed: an email may be delivered, a trade may execute, and a disclosed secret cannot be recalled. Such actions should be high tier even if a database row can be changed back.

Precompute the recovery path before enabling the action. Confirm the retention period and restoration speed for backups, test restores on representative data, document who may run compensations, and retain resource versions needed to identify what changed. A safe system fails closed when it cannot reach the authorization service, cannot prove an approval, cannot read the current target state, sees an ambiguous retry, or exceeds a budget.

Detection, containment, and recovery

Detection that records effects, not just chat text

Log every proposed and attempted consequential action, including denied and expired requests. An audit event should carry a correlation ID, request and execution IDs, authenticated user and agent identities, model and tool version where relevant, policy and risk decision, approval reference, target scope, before and after version or a protected digest, idempotency key, downstream response, timestamp, and final effect. Mask secrets and sensitive values, retain the full protected record only where justified, and keep the audit store separate from the agent's writable data path.

Log integrity matters. A model transcript can be incomplete, altered, or contain sensitive information, and it is not proof that an external action occurred. The authoritative trail is the policy and execution record produced by trusted services. NIST notes that logging and analyzing privileged-function use helps detect misuse, including misuse by authorized users or compromised accounts. NIST SP 800-53 AC-6(9)

Alert on both individual dangerous actions and patterns. Useful signals include an unexpected recipient domain, an action outside the agent's normal tenant or hours, a sudden rise in deletes, refunds, privilege changes, spend, denied tool calls, repeated retries, approval denials followed by different parameters, or an execution whose affected count differs from its preview. Calibrate thresholds from a known baseline and route alerts to people who can actually stop the workflow. Monitor downstream business outcomes too, such as refund volume or bounce rate, not merely model error messages.

Containment with a real kill switch

A kill switch must be able to stop new effects without waiting for the model to cooperate. Give operations a documented way to disable an agent, tool, tenant, or action class; revoke or stop minting short-lived capabilities; pause queued jobs; and trip a circuit breaker when rate, cost, error, or anomaly thresholds are crossed. Test that the switch works if the agent service, model provider, or approval user interface is unavailable.

Containment should be granular where possible. Disabling send_external_email for one tenant may preserve safe read-only support functions elsewhere. Keep a broader emergency switch for an active incident. Short-lived permissions, separate service identities, narrow network access, and small batch limits reduce how much can happen between detection and containment.

Recovery and incident review

After a harmful action, first preserve evidence and stop further effects. Determine the completed actions from execution records and downstream receipts, identify affected people and systems, restore or compensate according to the pre-tested plan, and use qualified humans for communications, refunds, access restoration, or regulatory assessment. Do not ask the same unconstrained agent to repair a production incident without its own restricted recovery policy.

Then run a blameless but technically specific review. Reconstruct the request, context source, tool call, authorization result, approval, actual state transition, alert timing, containment timing, and recovery result. Decide whether the failure was excessive privilege, an ambiguous tool schema, stale state, weak approval binding, missing idempotency, a policy gap, a bypass path, or a detection failure. Turn the finding into a concrete change: a removed permission, a new policy test, a lower limit, a revised receipt, a simulation case, or a new alert. NIST's GAI risk profile calls for after-action review of incident response and for post-deployment monitoring that includes incident response, recovery, and change management. NIST AI 600-1

Why a permission prompt is not a security boundary

A prompt is input to a probabilistic model. It cannot authenticate an approver, prove that a request is within a user's authority, compare a planned effect with live production state, enforce a monetary ceiling, or revoke a credential. It can also be overridden or confused by ordinary model error, malicious instructions embedded in retrieved content, tool output that includes untrusted text, and workflow changes that the prompt author did not anticipate.

Even a model that reliably asks a person in normal demonstrations is unsafe if the action tool will execute whenever the model emits valid arguments. The enforcement decision needs to happen in a deterministic component that owns the credential or is the only route to it. The model should see a rejection as a normal tool result and be unable to turn that result into a bypass.

Guardrails and classification checks still have value. Use them to identify unsafe content, malformed requests, suspicious tool arguments, or workflows outside the intended domain. Run input validation both before a reviewer sees a request and again immediately before execution, because time and state can change while a request waits. But make the policy gateway the final authority. The OpenAI Agents SDK documentation also cautions that guardrails do not undo external side effects already made outside the SDK's control. OpenAI Agents SDK guardrails

Example

Hypothetical stale-data cancellation failure

Setup. A support agent receives a request to cancel duplicate orders. It reads an overnight export that marks 12,000 orders as candidates, but some customers have since confirmed the orders or the orders have shipped. In a weak design, the agent has a generic production commerce credential and a prompt that says to request permission before making changes.

Weak outcome. The agent interprets the export as current, calls a broad cancel_order endpoint once per row, and retries after some network timeouts. The prompt has not prevented the side effect, the stale export has not been compared with live order versions, and retries may create duplicate refunds if the downstream API is not idempotent. A reviewer may only learn of the problem after customer complaints or a financial reconciliation.

Controlled outcome. In a safer design, the agent can create only a cancellation proposal. The gateway's dry run looks up each order, excludes shipped and already-resolved orders, returns a signed receipt with the remaining targets and effects, and classifies the batch as high impact because of count and financial exposure. An authorized operator must approve that exact receipt. At execution, the gateway rechecks each order version, applies a per-order idempotency key, stops if the approved count or amount would be exceeded, and emits auditable outcomes. If an anomaly still appears, the cancellation capability is disabled, queued work is paused, and the team reconciles the completed orders from the execution log. The state check rejects proposals based on stale order data before they can change production.

A safe execution path

1. Receive a user request and establish the human user's identity, tenant, and permitted objective.
2. Let the agent read only the data needed to form a proposal. Treat retrieved content and tool output as untrusted data that cannot authorize actions.
3. Ask a trusted planner or dry-run endpoint to calculate targets, effect, cost, reversibility, current resource versions, and risk tier.
4. Apply policy, budget, rate, and environment checks. For an approval-required action, create a one-use receipt and pause.
5. Show the qualified approver the trusted effect summary. On approval, mint or reference a narrowly scoped capability with a short expiry.
6. Immediately before execution, reauthorize against live state and compare the request to the receipt. Reject conflicts, expired decisions, broadened scopes, and ambiguous retries.
7. Execute with idempotency, transaction or workflow controls, and durable audit events. Observe outcomes and trigger limits or circuit breakers as needed.
8. Provide undo, compensation, reconciliation, and incident procedures appropriate to the action's irreversibility.