How should an AI application enforce per-user API budgets without adding noticeable request latency?

The core pattern

The durable accounting equation for one user and one reset window is:

available = hard_limit - settled_cost - active_reservations

Before sending a model request, the application calculates reservation_max, the maximum charge it has allowed the request to create. It then atomically performs this test and update:

if available >= reservation_max:
    active_reservations += reservation_max
    create reservation with a unique request ID
    authorize the provider request
else:
    deny or offer a smaller permitted request

After the request has a known usage result, a worker records the actual cost, subtracts the reservation, and adds the actual cost to settled_cost. If the actual cost is lower than the reservation, the difference becomes available again. The user-facing response does not wait for this bookkeeping, but the reservation is already durable before generation starts.

This is an authorization hold, similar in shape to a payment authorization, not a claim that the provider has billed the exact amount. Its strength comes from having one authoritative atomic mutation shared by all workers. A cache may make the user interface faster, but it cannot substitute for this mutation when the promise is a hard cap.

Why read then write overspends

Suppose a free user has spent $0.35 of their $0.50 monthly budget, leaving $0.15. Two browser tabs submit requests at the same time. Each request has a conservative maximum cost of $0.10.

With a naive read-then-write flow, both servers read $0.15 remaining, both decide that $0.10 fits, and both call the provider. Even if each post-call write is correct, the total can reach $0.55. The problem is the interval between the read and the write, not bad arithmetic.

An atomic reservation makes the same race safe. The first request changes reserved amount from $0.00 to $0.10 and succeeds. The second sees only $0.05 available and is denied or downgraded. If the first request later costs $0.06, reconciliation settles $0.06 and releases $0.04. The application does not need to predict $0.06 in advance to protect the $0.50 cap.

The same race appears with a $10 paid budget, only less often. More tabs, retries, streaming connections, background agents, and multiple serverless instances all compete for the same money. The check and reservation must be a single operation at the store that owns the balance.

Calculate a safe reservation

Use integer minor units, such as microdollars or nanodollars, throughout. Do not use binary floating point for money. Resolve a versioned price table before authorizing the request, then calculate a ceiling from request limits, not from average historical usage.

For a token-priced text request, a general formula is:

reservation_max =
  uncached_input_token_upper_bound × input_rate
  + cached_input_upper_bound × cached_input_rate
  + max_output_tokens      × output_rate
  + max_reasoning_tokens   × reasoning_rate, if the provider bills them separately
  + bounded tool or media charges, if enabled

The terms and units depend on the endpoint and provider. OpenAI explains that input, output, cached-input, and reasoning-token counts can differ, that output is not predictable from the input, and that field names vary between endpoints. OpenAI token accounting Use the provider's current model documentation or pricing page to populate rates, rather than putting prices in application code. OpenAI model documentation

For each allowed model and feature combination, configure a maximum output limit and any bounded tool budget. A user with $0.08 remaining can request a model only when its worst permitted request is $0.08 or less. Otherwise one of these must change before the call: its output cap, its selected model, its permitted tools, its budget, or the decision to reject it. If a provider exposes a billable component without a meaningful enforceable maximum, you cannot honestly promise a strict per-call monetary ceiling for that configuration.

Price tables need an effective date and an immutable ID. Store the price_table_id on the reservation and use it to calculate the actual cost for that request. A price change then affects new authorizations without rewriting history or leaving an investigator to infer which price was applied.

Do not reserve only average cost

Average-cost admission is fine for a deliberately soft plan, but it does not protect a hard budget. A short prompt can still generate an expensive output, and reasoning or tool use can make visible answer length a misleading proxy. The provider's response usage is the proper input to post-call reconciliation. The provider's organization-level Usage API also exposes usage by time bucket and can group data by model, project, API key, and user ID, but it is not a replacement for the application's own per-request ledger. OpenAI Usage API

OpenAI notes that Usage data and Costs data may have small reconciliation differences, and recommends its Costs endpoint or dashboard for amounts that reconcile to the invoice. Treat your local ledger as the real-time admission-control system, then reconcile it with provider billing data and correct any durable accounting discrepancy through an auditable adjustment. OpenAI Usage API

A minimal data model

Keep the budget window, reservation ledger, and price table separate. This design permits a quick hot-path mutation and a complete audit trail.

window_id must encode the reset policy, such as 2026-09 for a calendar-month plan or a tenant-local billing-cycle identifier. Store exact start and end timestamps, including the time zone policy, rather than relying on application-server midnight. A plan upgrade or downgrade needs an explicit rule: either create an adjustment in the existing window or apply the new limit only at the next window. Do not silently let a cached tier decide this.

Use an authenticated application user ID or an internal pseudonymous account ID, never a client-supplied arbitrary identifier. Budget records reveal behavior patterns even when they do not retain prompt text, so protect them as account data and keep prompts, completion text, credentials, and provider secrets out of the ledger.

Atomic authorization pseudocode

The following provider-neutral pseudocode shows the critical boundary. reserve() must run as a database transaction, a Redis function, or another single-writer operation. It cannot be implemented as two application-level calls named getBalance() and setBalance().

function reserve(user, request_id, request_plan, now):
    window = current_budget_window(user, now)
    existing = reservation_by_request_id(request_id)
    if existing exists:
        return existing.authorization_result

    price = active_price_table(request_plan.model, request_plan.endpoint, now)
    maximum = upper_bound_cost(request_plan, price)

    atomically:
        window = lock_or_compare_and_update(window)
        remaining = window.hard_limit_micro - window.settled_micro - window.reserved_micro
        if maximum > remaining:
            create rejected reservation(request_id, maximum, price.id)
            return DENIED

        increment window.reserved_micro by maximum
        create authorized reservation(
            request_id, window.id, maximum, price.id,
            state = "authorized", expires_at = now + settlement_grace_period
        )
        return AUTHORIZED

The application calls the provider only after AUTHORIZED. Then it queues a settlement job with the reservation ID and the provider response usage. The job is also idempotent:

function settle(reservation_id, reported_usage):
    actual = price(reservation.price_table_id).cost(reported_usage)

    atomically:
        reservation = lock(reservation_id)
        if reservation.state is "settled":
            return
        decrement window.reserved_micro by reservation.reserved_micro
        increment window.settled_micro by actual
        set reservation.actual_micro = actual, state = "settled"
        append usage_event

In a strict configuration, also reject a settlement that exceeds the reservation and flag it immediately. That should be impossible only if every billable dimension was bounded correctly. An overage means the request plan or price table was incomplete, a provider behavior changed, or the accounting code has a defect. Do not silently normalize it away.

Idempotency and retries

Generate request_id before authorization and reuse it for every retry caused by the same user action. Give the client a stable idempotency token or derive one from a durable job ID. A duplicate click, edge retry, worker retry, or reconnect must return the same reservation outcome rather than reserve money again.

Separate two retry questions:

• May the local authorization be retried? Yes, with the same request_id, because it is idempotent.
• May the provider call be retried? Only after determining whether the first attempt reached the provider. An unknown network outcome is not proof of zero token usage. Preserve the reservation while you query a provider request status when available, wait for a callback or usage record, or apply a documented conservative settlement rule.

If the provider definitively rejects the request before billable work, release the reservation through a recorded state transition. If the caller times out after the request may have been accepted, do not release the hold merely because the application did not receive a response. That is the budget equivalent of duplicating a payment after a timeout.

Streaming and cancellation

For a streamed response, authorize against the whole configured maximum before opening the stream. Rendering the first token quickly does not change the fact that later tokens can consume the remainder of the reservation.

When a user clicks Stop, send the appropriate cancellation or close action to the provider and mark the reservation cancellation_requested. Tokens generated before the cancellation reaches the provider can still be billable. For OpenAI streamed Chat Completions, stream_options: { include_usage: true } can return usage in a final chunk, but OpenAI specifically warns that an interrupted stream may not produce that final usage chunk and that its absence does not mean no tokens were used. OpenAI streaming usage guidance

Therefore, reconcile a completed stream from its reported usage, but keep an interrupted stream's reservation until an authoritative usage result, provider status, or conservative expiry policy resolves it. A low-latency product can tell the user that output stopped while its accounting remains pending. Releasing the full hold immediately makes a hard cap porous.

For long-lived or agentic sessions, divide the work into bounded turns. Reserve per turn, or reserve a session allowance and require each turn to spend from that allowance atomically. Do not allow an open-ended tool loop to draw from an unbounded monthly balance on the assumption that a later asynchronous worker will catch up.

Choosing the authorization store

All of these choices require a network round trip in a multi-instance system. The question is whether that round trip performs exactly one authoritative mutation near the request path, with the durability and recovery behavior your plan promises.

PostgreSQL defaults to Read Committed isolation, where a normal SELECT sees a snapshot at the start of that statement. That is why a separate read and later write does not protect the balance against a concurrent authorization. Use a row lock, a conditional update that includes the available-balance predicate, or a serializable transaction with retry handling. PostgreSQL transaction isolation

Redis is not correct merely because it is fast. A sequence of client-side GET, calculation, and SET still races. Redis documents that a server-side script or function executes atomically, while also warning that slow scripts block server activity. Keep the reservation operation short and use fixed keys and arguments. Redis programmability

For the Next.js and Supabase-shaped scenario, start by measuring where the reported 200 to 400 ms occurs: cold-start connection creation, connection pooling, network region, authentication, or the database query itself. A single Postgres stored procedure or RPC that reserves and writes the ledger can be enough. Introduce Redis only when measured hot-path latency or contention requires it, and keep a durable ledger plus recovery reconciliation either way. Do not promise a particular millisecond number without measuring the deployed region and traffic pattern.

Caches and stale reads

Cache a user's plan limit and displayed remaining budget for a short time if it improves the interface. Never use that cache to grant a hard-budget request. The only safe use is a hint, such as showing "about $0.12 left" or deciding to refresh a value before rendering.

If Redis is the authority, do not route the reservation script to a replica that can be stale. Handle an unavailable authority according to the plan's contract. A hard $0.50 free-tier cap generally fails closed: no new provider call until authorization succeeds. Failing open can be appropriate only when the plan explicitly permits a bounded overage, and then it is a soft limit, not a hard one.

Hard limits, soft limits, and service policy

Be direct about fail-open and fail-closed choices. If a reservation store outage causes a free request to proceed, the system has abandoned the hard cap for that period. That may be an acceptable paid-service availability tradeoff, but it should be recorded as an exception with a maximum emergency allowance, not hidden as normal behavior.

Rate limits are separate. A request-per-minute or token-per-minute limiter prevents a user from creating an abusive burst and protects provider quota. A budget limiter controls total allowed spend in a reset window. Use both, and reserve their capacity atomically if a request needs to pass both checks. A $10 monthly budget alone does not prevent ten expensive requests from arriving at once; a rate limit alone does not constrain month-long cost.

Reset windows and price changes

At authorization time, compute the current window from a trusted server clock, account timezone policy, and tier version. Avoid a background job that simply zeroes a mutable balance at midnight. An explicit new budget_window record prevents a delayed settlement from being posted to the wrong month and allows an old window to remain auditable.

Expire only reservations whose outcome has been investigated. An expiry worker should find reservations that are old, query or match any provider usage it can, settle known results, and hold or conservatively adjust unknown results according to policy. It should never delete an unknown request and assume it cost nothing.

When pricing changes, publish a new price-table version and use it only for new reservations. Monitor reservations that were created close to a price change, and reconcile provider invoice data separately from the real-time user budget. If discounts, committed-use tiers, currencies, taxes, or provider-managed tools make the invoice total different from simple public token rates, define whether the end-user plan is based on list-price credits or pass-through billed cost. That is a product and commercial rule, not a calculation the API response can settle by itself.

Operations and alerts

Monitor the allocator, not just total provider spend. Useful signals include:

• authorization allow and deny counts by tier, model, and window;
• reservation amount, settlement amount, and released difference;
• reservations stuck in authorized, unknown, or cancellation_requested states;
• attempts to settle above the reserved maximum;
• price-table version and missing-price errors;
• provider usage mismatches, ledger adjustments, and reconciliation lag;
• rate-limit denials, storage errors, and every fail-open exception;
• user-level low-budget thresholds and tenant-level aggregate spend thresholds.

Alert early enough to act, such as when a tier's remaining balance crosses product-defined thresholds or when unsettled reservations age beyond the normal provider-response window. The exact threshold is business-specific. What matters is that unresolved holds and settlement overages page an owner, because they are the events most likely to break the budget promise.

Keep an audit record for every authorization decision: user or account ID, window, request ID, price-table version, maximum reservation, final settlement, provider request reference, state transitions, and reason for denial or adjustment. This supports support tickets, refund decisions, fraud investigations, and incident recovery without retaining raw prompts.

Implementation checklist

1. Define whether each plan is hard, soft, or alert-only. Document the fail-closed or fail-open behavior.
2. Bound every billable request dimension: model, input size, maximum output, reasoning setting, tools, media, and retries.
3. Version the per-model price table and calculate money in integers.
4. Implement an idempotent atomic reserve operation backed by PostgreSQL, Redis, or a single writer.
5. Call the provider only after a reservation exists, then enqueue an idempotent settlement job.
6. Keep unknown timeouts and interrupted streams reserved until reconciled or conservatively resolved.
7. Add a separate rate limiter for bursts and provider quota protection.
8. Add dashboards and alerts for stuck reservations, pricing gaps, over-reservation, reconciliation differences, and exception paths.
9. Test concurrent same-user requests, duplicate client retries, worker crashes, provider timeouts, stream cancellation, price changes, and reset boundaries before enabling a hard budget.
10. Reconcile local ledger totals against the provider's reporting and billing data on a scheduled basis, using adjustments rather than rewriting settled history.