Can failed AI-agent traces be turned into useful fine-tuning data?

Treat a trace as a diagnosis record first

An agent trace is a record of a run: user request, prompt and policy version, retrieved context, available tools and schemas, tool-call arguments, tool outputs, timing, model response, evaluator result, and final outcome. It can reveal an important failure, but it rarely reveals the cause by itself.

For example, an agent that returns an incomplete answer may have reasoned poorly. It may instead have retrieved an obsolete document, been denied permission to the right system, called a tool with an invalid schema, exhausted a rate limit, received a timeout, or been scored by an evaluator with an incorrect expectation. Training on the visible final response without identifying the cause encourages cargo-cult repair.

Preserve the raw trace in a controlled incident store, then create a separate labelled record for analysis. The analysis record should name the task family, environment and tool versions, failure class, evidence, severity, whether the result is reproducible, suspected root cause, owner, and disposition. It should never assume that the model is at fault because the run ended badly.

This taxonomy prevents the most expensive mistake: changing model weights to compensate for an application defect. Current tool-calling documentation describes a multi-step flow in which the application executes tool calls and returns the output to the model. That means tool validation, permissions, retries, and side effects belong to application code, not to a fine-tune (OpenAI function-calling guide).

Decide whether a failure is a training candidate

Promote an analyzed failure to a candidate only when every answer below is yes.

1. Is the failure model-level? The correct retrieval, tool contract, identity, environment, evaluator, and task requirements were available and working, yet the model repeatedly made the wrong observable choice.
2. Is the correction independently verified? A deterministic test, trusted source, successful tool result, qualified human review, or accepted task outcome proves the correction. A plausible rewrite is not enough.
3. Is the behavior stable? The expected response will still be correct when prompts, tool names, permissions, or product policy change. Do not train a transient workaround into the model.
4. Is the record authorized and safe to use? The organization has a lawful and contractual right to use it for training, and secrets, personal data, customer content, credentials, internal URLs, and sensitive tool outputs have been removed or transformed under the applicable governance process.
5. Can it be evaluated without leakage? A related but distinct task, environment, and later time period can test the intended improvement. The exact corrected trace cannot be the only proof that the model improved.

If any condition fails, retain the trace as a debugging, monitoring, or evaluation artifact. That is still valuable. A well-labelled failure corpus can reveal which tools are brittle, which permissions confuse users, where retrieval is stale, and which task templates need clarification, without teaching any failure pattern to the model.

Convert verified failures into the right data form

The training objective determines the record shape. Do not concatenate a failed run and an improvised fix into a long transcript without deciding what behavior the model should learn.

Supervised fine-tuning, or SFT, trains against a preferred response. It is a fit when the team can state what the agent should have said or done. Preference optimization trains from relative labels, such as a rejected tool call and a preferred one for the same prompt. Direct Preference Optimization, or DPO, is one established preference-learning approach that uses a simpler classification loss rather than fitting a separate reward model, but it still relies on trustworthy preference labels (Rafailov et al., 2023).

For a current product-specific example, OpenAI's documentation describes SFT as examples of correct responses and DPO as a correct and an incorrect response for the same prompt. It also recommends an evaluation and prompt baseline before deciding to fine-tune (OpenAI model optimization guide). This is version-sensitive guidance. As of the verification date, the same page says OpenAI's fine-tuning platform is being wound down for new users, so teams should verify their provider's currently supported training route before committing to an implementation.

Curate before training

Remove sensitive material and establish rights

Agent traces are often more sensitive than ordinary chat logs. Tool arguments and outputs can contain access tokens, credentials, cookies, database rows, source code, customer identifiers, email addresses, financial data, internal hostnames, or privileged documents. A string scrubber is necessary but insufficient because a record can reveal identity or commercial information through context.

Use a documented intake process:

1. Restrict trace access to the people and systems that need it, then record every export into the curation pipeline.
2. Detect and remove or replace secrets and direct identifiers. Review samples for indirect identifiers and sensitive tool outputs.
3. Confirm consent, employment terms, customer agreements, data-processing terms, open-source licenses, and any data-residency restrictions for training use, not only for runtime logging.
4. Preserve an internal provenance link to the original incident without putting raw confidential content in the training file.
5. Apply a retention and deletion policy to raw traces, curated data, trained checkpoints, backups, and evaluation sets.

NIST's Generative AI Profile recommends documenting how provenance data interacts with privacy and security and considering anonymization and removal of personally identifiable information (NIST AI 600-1, p. 33). These are governance controls, not a substitute for legal advice or a finding that de-identification is always sufficient.

Deduplicate and weight the examples

Production logs are not a neutral sample. One faulty deployment, one popular task, or one malformed tool schema can create thousands of nearly identical failures. Training every copy can overweight a transient incident and degrade behavior elsewhere.

Deduplicate exact and near-duplicate tasks, tool calls, retrieved passages, and corrected completions. Group variants by task family, tool version, environment, customer or tenant where permitted, and root-cause label. Sample deliberately across successful and failed trajectories, easy and difficult tasks, languages, user groups, tools, and safety-relevant edge cases. Keep frequency as an analysis signal, not an automatic training weight.

Weight examples by verified impact and label quality. A small number of independently reviewed, high-consequence action-selection corrections may be worth more than a large collection of weak automatic labels. Conversely, do not repeatedly train on dramatic but rare failures until the system can define the desired general behavior and assess collateral effects.

Preserve observable evidence rather than private reasoning

An agent trace may include intermediate thought text, speculative notes, or hidden instructions. Those are not necessarily factual explanations of why the model acted, and they can expose sensitive information or encourage a brittle imitation of a particular reasoning style. Train on the observable contract instead: user input, permitted context, tool schema, tool call or response, verified outcome, and concise reviewer rationale where it helps define the correction.

For tool use, a strong training record might say that the required action is get_weather with a normalized location and date, that an unavailable result requires a clarification or fallback, and that no external action occurs without confirmation. It need not include a long internal monologue about tool selection.

Worked failed-tool-call example

Assume a support agent receives: “Show yesterday's failed payment count for the Acme workspace.” The available read-only tools are search_product_docs and query_payment_metrics. The latter requires workspace_id, an explicitly defined UTC start and end time, and a metric name. The agent calls search_product_docs and answers with a generic troubleshooting article. The evaluator marks the run failed.

The first task is diagnosis. If the customer name was not mapped to a workspace ID, this is an orchestration or identity-resolution gap. Add a safe resolve_workspace step that returns only workspaces the caller can access. If query_payment_metrics was missing from the tool list because the account lacks a data role, it is a permission problem. The correct behavior is an access-denied or handoff response, not a fine-tune that teaches the model to attempt a more privileged call. If the metric tool documentation incorrectly says it accepts yesterday while the API actually requires UTC timestamps, fix the tool contract and add schema tests.

Only after those controls work should the team test for a model-level selection failure. In a stable environment, the preferred behaviour could be: resolve the authorized workspace, normalize the requested period, call query_payment_metrics with the required fields, verify that the response is complete, and cite the as-of time. A successful recorded tool response or a deterministic metrics fixture verifies the target. This can become an SFT demonstration if the team needs a precise desired action sequence, or a preference pair if the wrong-doc-search action and the verified metrics action share the same context and differ only in the intended tool choice.

The same trace remains an evaluation case even if the team does not tune. Run it with mocked authorized, forbidden, timeout, and empty-result tool responses. The repaired agent should call the correct tool when authorized, request access or hand off when forbidden, use its bounded fallback when timed out, and avoid inventing a count when no data is returned. This demonstrates why prompt, tool, and orchestration repairs are often preferable to changing weights.

Keep training data separate from proof of improvement

Evaluation leakage is easy to introduce with traces. A corrected run can reveal the exact expected tool call, fixture output, evaluator phrasing, or task instance. If that same trace appears in training and final testing, a model may simply memorize a route or answer.

Use three separate collections:

Split by task family and time. Put related traces from the same user task, workflow, generated benchmark template, incident, tool-schema version, or retrieved-document version on one side of the boundary. Use a later time window for final holdout when tools, documents, or user behaviour evolve. For high-risk agents, include unseen tools, permission states, empty results, ambiguous requests, adversarial inputs, outages, and changed policies in the holdout rather than only successful happy paths.

Offline evaluation should replay deterministic fixtures or carefully captured tool responses, not make irreversible live calls. Measure task completion, correct-tool rate, valid-argument rate, schema compliance, source grounding, unnecessary calls, retry count, unauthorized-attempt rate, safe abstention, latency, and cost. Review error clusters manually because a high completion score can hide the same dangerous false success across many records.

The current OpenAI evaluation guidance likewise emphasizes testing against specified criteria and using representative test inputs, especially when evaluating or changing models (OpenAI evals guide). That provider's Evals product is scheduled for deprecation after the verification date, according to the same guide, so use the underlying evaluation practice and a supported toolchain rather than depending on a particular product surface.

Deploy the repair cautiously

An offline gain is not enough for an agent that calls tools or affects users. Begin with shadow testing: send production-like requests through the repaired system without exposing its answer or executing side effects, then compare its proposed actions with the existing system and qualified reviewers. Log the entire decision bundle, including tool availability, authorization outcome, retrieved context version, policy version, and final disposition.

Next use a narrow canary. Restrict the repaired model to a low-risk task family, read-only tools, small traffic share, and explicit stop conditions. Monitor new tool-call patterns, permission denials, unsafe attempts, omission rate, user corrections, latency, and cost. Require human approval for actions with financial, legal, medical, privacy, security, or safety consequences.

Maintain a fast rollback path. Version the model, prompt, tool schemas, retrieval index, policy, and evaluator together. A rollback may mean restoring a prior model, disabling a fine-tune, selecting a safer routing policy, or turning off a tool. Preserve the failed canary traces, label their root cause, and do not automatically feed them back into the next training job.

Common failure modes

Limits and viable alternatives

Fine-tuning is not the default remedy for an agent that fails. Clearer task contracts, smaller tool sets, better tool descriptions, structured arguments, retrieval filters, permission-aware orchestration, deterministic validators, and explicit safe fallbacks often solve the problem faster and with less regression risk. Current function-calling documentation notes that tools are application-provided functionality and that application code executes the model's requested action (OpenAI function-calling guide). Use that architecture to keep consequential controls out of model weights.

There are also cases where no curated correction should be created. If the task has no agreed answer, the environment cannot reproduce the outcome, the content is too sensitive to use, or evaluators disagree materially, keep the trace in an evaluation or incident set. Improve the task definition, human workflow, or system boundary first. For agents used in high-stakes domains, consult the relevant security, privacy, safety, legal, financial, or medical experts before any automated action or training use.