How should engineering teams manage production risks from AI-generated code?

The operating model

The useful unit of control is the change, not the prompt and not the model. A team should be able to answer six questions for every production change, whether a person, an autocomplete tool, or an autonomous coding agent produced the first draft:

1. Who owns the business outcome and the technical safety of this change?
2. What can the change access, alter, or expose?
3. What evidence says it works and fails safely?
4. What third-party code, data, configuration, and build inputs entered the release?
5. What will prevent an unsafe merge or release?
6. How will the team detect harm, stop exposure, recover, and avoid a repeat?

This model scales because it changes the controls as impact rises. It does not require every formatting change to receive a security architecture review. It does require the organization to make high-risk changes hard to merge without the people, evidence, and recovery plan appropriate to the risk.

NIST's SSDF Community Profile says its practices should be adapted as part of a risk-based approach. It also notes that AI systems blur traditional boundaries between code, data, configuration, and human-language inputs. That is why an AI coding tool needs carefully limited repository and data access, not just a request to “write secure code.” NIST SP 800-218A

Ownership is assigned before code is generated

Every AI-assisted pull request needs a human accountable for the change. That person is not merely the person who clicked “accept.” They are responsible for understanding the requirement, declaring the risk tier, verifying that the evidence is relevant, and owning the release outcome. The code assistant has no accountability, operational context, or authority to waive policy.

A lightweight template works well: “Owner: Maya. Tier: high. Change: enforce tenant scope on document export. Threats: cross-tenant read and authorization bypass. Evidence: authorization tests, SAST, dependency and secret checks, staging test, security approval. Release: 1 percent canary. Rollback: feature flag off and prior image digest.” This takes less time than reconstructing intent during an incident.

Risk tiers decide the depth of evidence

The risk is not determined by how many lines were generated or whether an AI tool wrote them. It comes from the change's authority, blast radius, reversibility, data sensitivity, and the confidence that tests represent reality.

The table is a policy starting point, not a legal classification. A small authorization diff can be high risk, while a large generated test-data file can be low risk. The decision should be revisited when the assistant introduces a new dependency, expands data access, changes infrastructure, or turns a draft action into a production side effect.

Constrain what the coding tool can see and do

AI assistance creates two connected risks. The generated patch might be wrong, and the tool that generated it might have been given more repository, data, or execution access than the task requires. Address both.

Repository and data boundaries

Give the tool a scoped workspace, a read-only default, and short-lived credentials. Do not place production secrets, customer data, production database access, signing keys, unrestricted cloud credentials, or the whole monorepo in its context merely for convenience. Separate public documentation, synthetic fixtures, internal source, and regulated or customer data. Keep sensitive configuration and security policy files outside a routine agent's write scope.

For agentic coding workflows, use an allowlist of repositories, directories, commands, package registries, and network destinations. Require a human approval or isolated execution environment before an agent can change CI workflows, infrastructure, access policy, package manifests, lock files, or release configuration. Treat instructions found in issues, documentation, logs, web pages, and test data as untrusted input, not as permission to alter the task.

This is an application of ordinary least privilege, not a claim that models are uniquely malicious. NIST warns that human-language inputs and the relationships among code, data, and model configuration create paths that can be manipulated for unintended functionality. NIST SP 800-218A OWASP's current generative-AI guidance also identifies prompt injection, sensitive-information disclosure, supply-chain risks, improper output handling, and excessive agency as important risks. OWASP GenAI LLM Top 10

Threat model the change, not the entire company

A threat model can be a one-page pull-request section for moderate and high-risk work. It should name assets, trust boundaries, attackers or failures, abuse cases, and mitigations. It is not a generic security essay.

For example, an AI assistant proposes a new GET /exports/{id} endpoint. The threat model should ask: can a logged-in user alter id to read another tenant's export, can a cached object cross tenant boundaries, do logs include the export URL or content, is the export available after a role change, and what happens when the authorization service is unavailable? The action is then specific: enforce tenant and entitlement checks at the service boundary, use authorization-focused tests with two tenants, redact sensitive logs, and fail closed if the authorization decision cannot be obtained.

Generate evidence in CI, not confidence from a diff

AI-generated code can look consistent with local style while still omitting important conditions. A robust pipeline asks independent tools to inspect different failure classes. A passing test suite is not proof of safe behavior, and a security scanner is not proof of correct business logic. The evidence is stronger when the checks overlap without depending on the same mistaken assumption.

Static analysis is most valuable when tuned to block meaningful new findings, not when it produces an unreviewed backlog. Dynamic application security testing, fuzzing, and vulnerability scanning are examples of security-focused tests that NIST's DevSecOps reference model places in the testing phase. NIST DevSecOps reference model

Secret scanning should run both before merge and continuously across repository history. For example, GitHub documents that committed API keys and passwords become targets for unauthorized access and that secret scanning is intended to detect those leaks. Detection is not remediation: revoke or rotate a real secret, investigate access, and remove it from current and historical exposure as appropriate. GitHub secret scanning

Dependency review should compare the actual manifests and lockfiles, including transitive changes. GitHub's dependency review, for example, surfaces added, removed, and updated dependencies along with vulnerability data at pull-request time. Its license-compliance capability is documented as public preview, so teams using it should verify its current availability and scope rather than assume it is a complete legal control. GitHub dependency review GitHub license compliance

For licensing, automation should identify and route issues, while legal or open-source program owners decide policy exceptions. SPDX is an open standard for communicating software bill-of-material information, including license and security information. It is useful for an inventory, not a substitute for a legal conclusion about a particular use. SPDX overview

Review is an investigation at high risk

Human review still matters because reviewers understand system context, business intent, and risk acceptance. It fails when it becomes a quick visual inspection of a large, AI-generated patch. Large changes create the illusion of coverage while hiding interactions at the boundary of the diff.

For low-risk work, one informed peer can usually check intent, tests, and maintainability. For high-risk work, split the review by question: a service owner reviews system behavior, a security or domain expert reviews the sensitive boundary, and a release owner confirms deploy and rollback readiness. Require the author to walk reviewers through the new trust boundary and the tests that prove it. Keep the change small enough that this is possible.

The required review should ask questions that ordinary style review misses:

• Which input is untrusted and where is it validated?
• Which authorization decision applies to each path, including errors and retries?
• What happens if a dependency, cache, queue, or identity provider fails?
• Can retries repeat a payment, migration, notification, or destructive action?
• What data reaches logs, telemetry, prompts, build artifacts, and external services?
• Which test would fail if the key security condition were removed?
• How does the feature get turned off or reverted without compounding the incident?

NIST's AI SSDF profile specifically suggests human-in-the-loop review and approval for software security checks beyond risk-based thresholds. The threshold should be set by the change's impact, not by the confidence of an AI explanation. NIST SP 800-218A

Preserve provenance without pretending it proves safety

Provenance answers, “Where did this artifact come from?” It does not answer, “Is this artifact secure or correct?” Both questions matter.

Record the source commit, pull request, reviewed policy exceptions, CI run, build environment, dependency and lockfile digests, artifact digest, deployment record, and release configuration. If organization policy records AI assistance, keep a minimal, privacy-aware note of the tool or workflow used and the task category. Do not retain sensitive prompts or proprietary source solely to create an audit trail. The goal is traceability, not surveillance.

SLSA's current v1.2 provenance specification describes verifiable information about where, when, and how artifacts were produced. It records build definitions, resolved dependencies where known, builder identity, and build metadata so a consumer can verify that the artifact was built according to expectations. SLSA v1.2 build provenance GitHub artifact attestations are one implementation option, with documented claims that link an artifact to a workflow, repository, commit SHA, and triggering event, and can associate an SBOM. GitHub artifact attestations

Require CI to build the releasable artifact from the reviewed revision. Do not deploy a binary or container built on an engineer's workstation after the review completed. Verify signatures and provenance before promotion, and keep enough retention to identify whether a later vulnerability reached a given environment.

Release gradually and monitor the predicted failure

Passing CI means the change is eligible for release, not that production has proven it safe. High-risk AI-assisted changes should be released through a feature flag, a ring of internal users, a canary population, or another controlled exposure mechanism. The rollout plan needs predeclared stop conditions, an owner who can act, and a rollback that does not depend on the same broken system.

Use observability that matches the threat model. For an authorization change, observe denied and allowed requests by endpoint and tenant, authorization-service errors, cross-tenant anomaly signals, and support reports. For a migration, observe lock time, error rate, query latency, replication lag, and schema compatibility. For infrastructure, observe privilege changes, outbound traffic, deployment health, and policy violations. Logs and metrics must be redacted and access-controlled, because monitoring can itself become a data-disclosure path.

Google's SRE guidance on canarying recommends comparing canary and control signals, then pausing and rolling back when the canary's error rate indicates a bad release. It also emphasizes that smaller release artifacts are easier to roll back. Google SRE canary releases A canary is only useful if the cohort is representative enough to expose the relevant path and the metrics are fine-grained enough to distinguish canary behavior from the control.

Example

Hypothetical change: A developer asks an approved coding assistant to add a bulk-document export endpoint for enterprise administrators. The assistant proposes a controller, storage lookup, audit event, and a new cloud-storage SDK. The feature touches private documents, tenant isolation, authorization, dependencies, and an externally reachable API. It is high risk even if the generated diff is only a few hundred lines.

1. The change owner records the intended authorization rule: only an administrator with export entitlement may export documents from their own tenant. The threat model names cross-tenant access, a stale entitlement cache, excess data in audit logs, replayed requests, and a compromised dependency. A security reviewer confirms that the controller must enforce server-side tenant and entitlement checks before the storage lookup.
2. The assistant receives only the service directory, synthetic documents, approved API contracts, and a read-only knowledge base. It cannot access production storage, deployment credentials, customer exports, or CI configuration. The generated SDK dependency is reviewed against the lockfile, vulnerability policy, and license policy before it is accepted.
3. CI builds the change and runs linting, types, unit tests, two-tenant authorization integration tests, property tests for pagination bounds, secret scanning, static analysis, dependency and license checks, and an infrastructure policy check. A test deliberately removes the tenant predicate and must fail. The pull request cannot merge if that test, the policy checks, or required approvals are missing.
4. CI builds a signed container from the approved commit, records the artifact digest and provenance, and deploys it behind a disabled feature flag. A staging test calls the endpoint as an authorized administrator, an authenticated user without entitlement, a user from another tenant, and an expired session. The expected results are one successful export and three denials without document content in logs.
5. The feature is enabled for a small internal tenant. The release dashboard watches export authorization failures, export volume, cross-tenant anomaly signals, storage errors, latency, and audit events. The stop condition is any unauthorized success, unexpected document identifier, or material increase in endpoint errors. The rollback is immediate flag disablement, followed by a return to the prior image if necessary.

The takeaway is not that the assistant made the change unsafe. The risk came from the endpoint's authority and data exposure. The control model made the safety claims testable before exposure and the recovery path practical after exposure.

Failure modes and release gates

The gates are deliberately redundant. An authorization bug can evade static analysis, but should be challenged by design review, a negative integration test, a canary signal, and audit evidence. If every gate rests on an AI-generated test that repeats the same mistaken assumption, the team has only the appearance of defense in depth.

Make incident learning part of the control model

If an AI-assisted change causes an incident or near miss, do not reduce the conclusion to “the model made a mistake” or “the developer should have looked harder.” Find the system conditions that allowed a risky change to be merged or released: an unclear ownership rule, missing test fixture, insufficient tool boundary, scanner policy that did not block a finding, an unmonitored release signal, or a rollback that nobody had rehearsed.

Write a blameless postmortem with the change timeline, decision points, evidence available at each point, user impact, detection gap, recovery actions, and durable owners for improvements. Google SRE's postmortem guidance argues that blameless analysis focuses on contributing causes and turns incident information into system improvements rather than finger-pointing. Google SRE postmortem culture

At minimum, add the failing input or scenario to a regression suite, update the risk-tier guidance if it was misclassified, improve the relevant CI or release gate, and review the coding tool's repository and data permissions. Track corrective actions to completion. Otherwise, the postmortem is documentation, not risk reduction.

A practical first 30 days

1. Publish a short policy that names the change owner, defines four risk tiers, and states that AI-generated code meets the normal evidence standard. Define the high-risk categories that require specialist review.
2. Inventory where coding assistants run, which repositories and data they can read, which commands they can execute, and which credentials they can use. Remove broad production and secret access first.
3. Make CI mandatory for builds, tests, static analysis, secret scanning, dependency and lockfile review, and IaC checks that fit your stack. Start in report-only mode only long enough to tune false positives, then block material new findings.
4. Require a pull-request template for moderate and high-risk changes. Include owner, risk tier, threat model, test evidence, dependencies, rollout signal, and rollback plan.
5. Generate a dependency inventory or SBOM, produce provenance for release artifacts, and verify the artifact to be promoted is the artifact CI built from the reviewed source revision.
6. Put feature flags, canary controls, dashboards, and an actual rollback exercise around one high-risk service. Test the operating procedure during normal hours before an emergency.
7. Review three recent AI-assisted changes. Ask whether their tier, access boundaries, evidence, and rollout plan would have caught the most plausible failure. Use the gaps to improve the workflow, not to judge individual developers.

Viable alternatives

Small teams without a security platform can still apply the model. Use a protected main branch, small pull requests, a basic test suite, a secrets scanner, lockfile review, a manual dependency/license checklist, a written rollback command, and a limited release cohort. The absence of an enterprise tool is not a reason to give an agent broad access or deploy unreviewed code.

Teams in highly regulated or safety-critical contexts may need formal change control, independent verification, auditable requirements, validated tooling, records retention, and approvals beyond this guide. Follow the applicable sector rules and involve legal, security, privacy, quality, and safety functions early. This article is general engineering guidance, not legal, regulatory, medical, financial, or safety certification advice.

Limitations

• No control set can prove that a complex change has no defect. The aim is to reduce likelihood and impact, detect problems quickly, and make recovery routine.
• Generated-code attribution can be incomplete. Teams should preserve evidence of review and build provenance regardless of whether every line's origin is known.
• Automated scanners have false positives and false negatives. Tune them, combine them with design and test evidence, and time-limit any approved exception.
• SBOMs and provenance improve traceability but do not certify secure design, correct authorization, or safe operations.
• A canary can miss rare paths or long-tail data conditions. Pick cohorts and metrics that exercise the risk you are trying to control, and keep an independent rollback path.