What should an AI-agent transaction log record so runs can be audited, resumed, and repaired?

The design in one sentence

Create one ordered event stream per agent run, plus referenced artifact storage, and make every state transition and attempted external effect append an immutable event before and after it happens.

The word "event" should mean a fact with business meaning, not a line of unstructured debug text. Good examples are RunStarted, PolicyEvaluated, HumanApprovalGranted, RefundIntended, PaymentProviderObserved, RetryScheduled, CheckpointCreated, CompensationRequested, and RunCompleted. A correction adds a new fact that refers to the earlier one. It does not silently edit history.

An event-sourced store is an established pattern in which state changes are recorded in an immutable, append-only, chronologically ordered event store. Replaying the stream can reconstruct a point-in-time view. AWS event sourcing guidance For agent systems, the ledger normally records the agent execution and its controlled side effects. It does not require every unrelated business table to be redesigned as event sourcing.

Keep five complementary records distinct and linked

OpenTelemetry is valuable for connecting traces, metrics, logs, and events with shared semantic conventions. It describes events as named occurrences at a meaningful point in time, including state transitions and outcomes. OpenTelemetry event conventions Use the trace ID on ledger events, but do not make sampled or retention-limited telemetry the only source of recovery truth.

A workflow engine can keep a current step, retry state, revision, and execution history. Google Cloud Workflows, for example, exposes an execution state, workflow revision, step history, and configurable detailed history. Google Cloud Workflows execution results That is useful durable state. The run ledger should additionally distinguish an intended payment request from a confirmed payment-provider outcome and retain the policy, approval, and artifact references needed to explain the decision.

The row in the first column is intentionally not a substitute for the second. A local database commit cannot atomically cover a payment API, an email provider, a model call, and a human approval. Agent workflows are distributed processes, so recovery relies on idempotency, observations, and compensating actions rather than pretending there is one global transaction.

The minimum event envelope

Give every event a schema version and enough identifiers to make ordering and causality explicit. A global timestamp is not a sufficient ordering mechanism when clocks drift and work runs in parallel.

{
  "event_id": "evt_01J...",
  "event_type": "tool.effect_observed",
  "event_schema_version": 1,
  "run_id": "run_01J...",
  "step_id": "step_refund_2",
  "attempt": 1,
  "stream_sequence": 42,
  "occurred_at": "2026-09-01T14:31:08.412Z",
  "recorded_at": "2026-09-01T14:31:08.518Z",
  "actor": {"type": "agent", "id": "refund-agent", "identity_version": "2026-08-15"},
  "causation_event_id": "evt_01J_intent",
  "correlation_id": "support-case_8472",
  "trace_id": "4bf92f...",
  "artifact_refs": [
    {"kind": "prompt_render", "sha256": "...", "storage_ref": "artifact://..."},
    {"kind": "policy_snapshot", "sha256": "...", "storage_ref": "artifact://..."}
  ],
  "data": {
    "tool": "payment.refund",
    "tool_version": "2026-08-01",
    "idempotency_key": "refund:order-1042:line-1",
    "provider_effect_id": "re_123",
    "outcome": "succeeded"
  },
  "previous_event_hash": "...",
  "event_hash": "..."
}

The envelope is illustrative. It does not say that every field belongs in every event. Keep the stable envelope compact, place large or sensitive content in an access-controlled artifact store, and include content hashes and immutable references in the ledger. A hash shows that a retrieved artifact matches the recorded bytes. It is not a safe substitute for access control, and it is not automatically anonymous when the original value is guessable.

What each run should record

Record what was decided separately from why it was accepted. A model may propose refund_amount: 40.00; a deterministic policy evaluator or a human approval should record whether that amount was allowed and which policy snapshot supported it. Avoid treating a model's prose explanation as a reliable audit reason where a policy rule, order record, or approval is available.

Intention before effect and observation after it

The critical event pair for a side effect is:

1. effect_intended: the validated action, target, constrained amount or arguments, approval reference, and idempotency key are durably appended.
2. effect_observed: the system appends confirmed success, confirmed rejection, known failure, or an explicitly unknown outcome after contacting the external system.

Between those events, a worker can crash, the network can fail, or the remote system can process the request after the local timeout. Do not rewrite effect_intended as failed just because the caller lost its connection. Preserve the uncertainty and run a recovery step that queries the remote system with the idempotency key or an equivalent effect reference.

If the external system cannot support idempotency or status lookup, the ledger should make that limitation explicit. The system then needs a conservative manual-review state or a compensated business process. It cannot honestly guarantee exactly-once delivery by writing more log lines.

Retries need a typed policy. Record the error class, retryability decision, attempt count, backoff schedule, and the same idempotency key if the retry is meant to be the same logical effect. Google documents distinct retry behavior for idempotent and non-idempotent workflow steps, including bounded attempts and backoff. Google Cloud Workflows retries Apply that distinction at the tool boundary, not as a generic instruction to retry all agent failures.

A detailed refund example

Example

This hypothetical support agent handles an order-delivery complaint. The user asks for a refund for order O-1042. The system may look up the order and shipping status, propose a refund, require policy approval for higher amounts, call the payment provider, and notify the user.

1. Start and bind the evidence. RunStarted records the authenticated support case, an input hash, user data classification, run ID, current policy revision, agent workflow revision, and model configuration. OrderLookupIntended references the allowed order ID. OrderLookupObserved records the order and delivery-status artifacts by encrypted reference and content hash.
2. Make a proposal, not a payment. DecisionProposed records a structured candidate refund amount and links the policy and order artifacts. PolicyEvaluated records whether the amount is within an automatic threshold. If it is not, ApprovalRequested creates a task for a supervisor. The supervisor's ApprovalGranted event states exactly which order, amount, and expiry it authorizes.
3. Durably intend the refund. Before the network call, the system appends RefundIntended with idempotency_key = refund:O-1042:line-1, the approved amount, the payment-provider tool version, and the approval event ID. The state projection now says refund_pending_observation.
4. Crash at the dangerous point. The worker sends the provider request, then crashes before it receives a response. There is no RefundObserved event. A new worker sees an unobserved intention, not a completed refund. It does not create a new refund with a new key.
5. Recover by observation. The recovery worker calls the provider's status endpoint using the recorded key. If it finds refund re_123, it appends RefundObserved with that external effect ID and moves the state to refund_confirmed. If it finds no effect and the provider documents safe reuse of the key, it retries with the same key. If the provider cannot answer, the state becomes refund_unknown_requires_review; the hold remains visible to a human.
6. Notify and finish. CustomerNotificationIntended uses a second idempotency key. CustomerNotificationObserved records the provider message ID. RunCompleted records succeeded only after the required outcome and notification states are known.

The ledger tells an operator why the refund was proposed, who approved it, which policy applied, whether the provider effect happened, and why recovery took its path. A trace may show the failed HTTP span, but the trace alone does not prevent a duplicate refund. A checkpoint may say "payment step," but without the intended event and idempotency key it cannot safely distinguish a request that never left from one that succeeded remotely.

Repair is a new business action

Suppose an operator later discovers that the refund amount should have been lower. Do not edit RefundObserved out of the ledger. Append RepairOpened, link the evidence for the error, and append a CompensationIntended event for the permitted remedy, such as a recovery charge, account credit adjustment, or an exception process. Then record the observed result.

Compensation is not a universal undo button. A sent email cannot be unsent, a refund may be legally or commercially irreversible, and a model may have exposed information before a correction. The ledger should record the residual impact, the owner, and the reason the original action was superseded. This is how an audit trail remains honest when the world cannot be rolled back.

Checkpoints make replay practical

A long agent run should not reconstruct every object by re-running every model call. Periodically append CheckpointCreated with a state-machine snapshot or artifact reference, the event sequence it covers, a snapshot schema version, and a hash. The current workflow state is then a projection: start from the newest compatible checkpoint and apply later events in sequence.

The checkpoint itself is not the audit record. It is an acceleration structure. If a checkpoint is corrupt or its schema becomes unsupported, rebuild it from the event stream and referenced artifacts where retention allows. Keep checkpoint migration code versioned and test it against historical streams.

There are three different kinds of replay:

Language-model generation is probabilistic and model serving can change. Preserve the original structured decision output, tool arguments, and artifact references as evidence. Do not claim that replaying the prompt will reproduce the original completion. Where exact reconstruction is required, replay the recorded decision and recorded tool results through deterministic code, not a fresh model call.

Tamper evidence and access controls

Append-only is an application behavior, not proof that a privileged administrator cannot alter history. For a ledger with meaningful audit requirements, combine several controls:

• give the append service a narrowly scoped write identity and make ordinary application identities unable to update or delete events;
• use immutable storage or retention controls for exported ledger segments and artifacts, with separate security administration;
• hash each event over its canonical bytes and the prior event hash in its stream, then periodically anchor a signed digest outside the main writer's control;
• monitor missing sequence numbers, failed append attempts, digest validation failures, and break-glass access;
• restrict reading of raw artifacts more tightly than reading event metadata, and record every sensitive artifact access.

AWS CloudTrail provides a useful, not agent-specific, illustration. Its log-file validation uses hashes and signed digest files that reference prior digests, so modification or deletion can be detected. AWS CloudTrail log-file integrity validation A hash chain detects unauthorized alteration after the fact. It does not prevent an authorized but malicious writer from recording false information, nor does it make a flawed agent decision correct.

Use roles that reflect real duties: application workers can append narrowly scoped events; recovery workers can append settlement events; operators can read metadata and request approved repairs; auditors can read immutable exports; only a tightly controlled security role can perform break-glass access. Log the access decision itself. Do not put unrestricted prompts, tool arguments, tokens, customer records, or model output in a broadly searchable tracing system.

Retention and deletion are design requirements

An audit need, privacy obligation, and operational debugging need can have different retention periods. Classify events and artifacts before storing them.

Deletion and append-only history can conflict. A practical pattern is to append ArtifactRedacted or DeletionCompleted with the reason and scope, then delete or cryptographically destroy the encrypted payload according to the policy. The historical event remains as a non-sensitive record that a deletion occurred, but should not retain the deleted content. Whether that meets a particular legal requirement depends on the jurisdiction, contract, and data category. Obtain privacy and legal review for regulated personal, financial, health, or child data.

NIST's SP 800-92 Rev. 1 Initial Public Draft treats log management as part of audit and accountability, monitoring, and incident response planning. It is draft guidance, not a final publication. NIST SP 800-92 Rev. 1 Initial Public Draft Apply the same operational discipline to agent artifacts: retention schedule, capacity plan, access review, incident procedure, and tested restoration.

Build it in stages

1. Define the agent's state machine and enumerate every effect that can change money, data, access, customer communication, or external systems.
2. Assign a run ID, step ID, event schema, and idempotency key strategy before wiring model calls or tools.
3. Implement an append-only event table or event store with optimistic concurrency per run stream. Add a materialized state projection for fast status reads.
4. Add the intention and observation pair around each external effect, then test crashes between every pair of steps.
5. Store prompts, documents, responses, and tool payloads as classified artifacts with hashes and controlled references, not indiscriminate log fields.
6. Add checkpoints, recovery workers, bounded retry policies, and an escalation state for ambiguous external outcomes.
7. Correlate trace IDs and metrics with ledger events, but retain the ledger even when telemetry is sampled or expires.
8. Add digest anchoring, access review, retention jobs, deletion workflow, and restore drills when the risk justifies them.

Failure modes to test

Test the cases that prose diagrams tend to hide: duplicate user input, duplicate queue delivery, worker crash after external send, provider timeout after success, provider rejection, out-of-order callback, a changed tool schema, a changed policy revision, a missing artifact, a failed checkpoint migration, expired retention, and a manual repair that itself fails halfway through.

For each test, inspect the event stream and the state projection separately. The projection should show a safe current status. The stream should explain why it has that status and show that a retry or repair will not repeat an already confirmed side effect.