Should AI coding agents build healthcare software, or remain in a supervised support role?

The governing rule is risk and intended use

Treat an AI coding agent as a development tool inside a healthcare software process. It does not inherit authority to determine a product's intended use, privacy posture, clinical validity, regulatory status, or readiness for release. Those are accountable decisions made by the organization and qualified people with the relevant engineering, security, clinical, quality, and regulatory roles.

The same generated function can have radically different consequences in different deployments. An appointment-calendar component may be administrative. A function that uses the same calendar data to recommend who should receive urgent care becomes a clinical decision function. A chatbot that reformats a clinician-authored note differs from one that tells a patient to change medication. Look at the actual claims, workflow, user, data, output, and foreseeable misuse, not the label "assistant" or a disclaimer pasted onto the screen.

In the United States, FDA describes its software policy as function-specific and independent of the platform on which a software function runs. FDA's current Clinical Decision Support guidance distinguishes some non-device clinical decision support functions from functions that remain medical devices, including functions intended for patients or caregivers. FDA Clinical Decision Support guidance This is general regulatory information, not a product classification determination. A manufacturer should seek qualified U.S. regulatory and legal advice for its actual intended use and claims.

Use a four-tier development posture

This table is a decision rule, not a legal classification chart. A low-risk administrative feature can still expose sensitive records or cause a patient safety incident through bad identity matching or authorization. Conversely, an application with a mixed feature set should be assessed function by function. FDA's current FAQ states that a product can contain both non-device functions and device functions, and that the device function may be assessed with regard to the other functions. FDA CDS FAQs

The safe default is to begin at the highest plausible risk tier, then reduce controls only with documented evidence. Do not start with an agent-generated prototype and try to add the evidence after people rely on it.

The scheduling and diagnostic contrast

Example

Consider two hypothetical services built by the same team.

The first is a scheduling assistant. A staff member asks it to find an available primary-care appointment, prepare a message, and route the request to a clinic queue. The assistant has access only to appointment slots and the minimum patient identifier needed for routing. It does not infer urgency, diagnose a condition, prioritize one patient over another based on health data, or give medical advice. An AI coding agent may generate a form component, a mock scheduling adapter, or authorization tests in a sandbox. Engineers still review access control, patient matching, audit events, error paths, backup recovery, and the exact integration before release.

The second is diagnostic software. It accepts a chest image and patient details, labels a condition as likely, and presents a treatment or escalation recommendation. The output can affect care and may be acted on even if a clinician is nominally present. A fluent explanation does not establish clinical validity, adequate sensitivity across populations, safe usability under time pressure, or appropriate regulatory status. An AI coding agent may assist with a tightly scoped internal task, but it must not determine the clinical algorithm, choose acceptance criteria, fabricate test evidence, approve a model update, or deploy the system.

The takeaway is that both products need secure engineering, but the diagnostic software also needs a much stronger evidence and lifecycle case. Intended use and possible harm change the controls. A clinician clicking "accept" is not a substitute for showing that the system is fit for the clinical use being claimed.

Protect health information before generation begins

In the United States, HIPAA applies to covered entities and business associates, not automatically to every software project. When a covered entity or business associate uses a cloud service provider to create, receive, maintain, or transmit electronic protected health information, HHS says a HIPAA-compliant business associate agreement is required, along with compliance with the HIPAA Rules. HHS guidance on HIPAA and cloud computing

That makes provider terms and configuration a design decision, not a checkbox. Before allowing a hosted coding agent, model API, repository connector, issue tracker, or observability product to see protected health information, confirm the organization has approved the specific product tier, data-use terms, retention setting, region, access path, business associate agreement where applicable, subcontractor arrangement, and incident obligations. A generic consumer account, public pastebin, or unapproved browser extension is not an acceptable route for clinical records, even if a developer believes the data is encrypted or de-identified.

Use data minimization as the default.

• Give coding agents synthetic fixtures, schema-only examples, and de-identified cases whenever possible.
• Redact direct identifiers, free-text notes, images, and hidden document metadata before a task reaches an external system.
• Keep production credentials, signing keys, database dumps, full logs, and patient record exports out of agent context.
• Use separate development, test, staging, and production accounts. The agent should not possess a production credential during generation or test execution.
• Record which tool received which data class, who approved it, and how long it remains available.

The HIPAA Security Rule requires regulated entities to protect electronic protected health information with administrative, physical, and technical safeguards, and to obtain appropriate assurances from business associates. HHS Summary of the HIPAA Security Rule Those safeguards apply to the system around the agent as well as the finished application. They do not make a developer prompt safe by themselves.

Give the agent less authority than the developer

An agent should begin with read-only repository access and a disposable local or cloud sandbox. It may propose a diff and run permitted unit tests against synthetic data. Escalate privileges only for a specific task, to a known path, for a limited time, with a responsible human able to inspect the result.

Sandboxing reduces blast radius but does not prove generated code is safe. Least privilege also has to apply to connected tools. An agent that can read an issue tracker, call an internal knowledge base, browse the web, or invoke a database tool can be influenced by untrusted content or leak information through its outputs. Treat tool descriptions, retrieved pages, tickets, and code comments as untrusted input. Keep network access, tool allowlists, command logging, and approval boundaries explicit.

Build requirements traceability before code volume grows

For every safety, privacy, security, and clinical requirement, create an identifier and maintain a trace from source requirement to design control, implementation, test, evidence, release decision, and post-release monitoring. A simple traceability record can be more valuable than a long prompt history.

AI-generated code enters this chain like any other supplied component. Record agent and model configuration, task packet, prompt or a controlled prompt reference, changed files, tool calls, dependencies added, tests run, and reviewer disposition. Do not make a model's natural-language explanation the source of truth for traceability. The approved requirement and verified evidence are the source of truth.

For U.S. finished medical-device manufacturers, FDA's Quality Management System Regulation became effective on February 2, 2026 and incorporates ISO 13485:2016 by reference as the foundational quality management system framework. FDA says the regulation applies to finished device manufacturers commercially distributing medical devices, and includes risk management and design and development controls. FDA QMSR A coding agent can accelerate individual artifacts, but it cannot replace the established quality system or the manufacturer's responsibility for those records.

Secure development and independent evidence

Use a secure development lifecycle that starts before the first generated change and continues through decommissioning. At minimum, maintain threat models, architecture review, least-privilege design, dependency policy, software bill of materials where applicable, secure configuration, code review, static analysis, secret scanning, vulnerability management, test environments, and incident playbooks.

For FDA-regulated U.S. cyber devices, the statutory and premarket context can require cybersecurity information, including processes for vulnerability management and a software bill of materials. FDA's February 2026 final cybersecurity guidance describes a Secure Product Development Framework spanning design, development, release, support, and decommissioning. FDA medical-device cybersecurity guidance This does not mean every scheduling tool is a cyber device. It does mean that a connected clinical device needs lifecycle cybersecurity evidence, not a one-time scan.

Tests must be capable of disproving a claim. Generated unit tests that mirror generated implementation do not establish safety. For an administrative record system, include tests for authorization, patient identity matching, consent state, data isolation, failure recovery, audit logging, backup restore, and malformed integration inputs. For clinical functions, add protocol-driven verification and validation, clinical performance evaluation, representative data assessment, and independent review that matches the intended use and risk.

Independence matters when an error can harm a patient. The engineer who used the agent should not be the only person deciding whether the agent's work is correct. Involve a security reviewer for access and data flows, a clinician or domain expert for clinical workflow and evidence, a safety or quality owner for risk controls, and a release owner for the final deployment decision. Their authority and acceptance criteria must be documented before the result is convenient to approve.

Design against bias, misuse, and poor usability

Clinical risk does not disappear because an output has a confidence score or a human can override it. A user may defer to an apparently authoritative recommendation, misunderstand uncertainty, lack time to verify it, or receive an output that performs differently for their population or setting.

Define the intended users, environment, patient populations, input quality, language, accessibility needs, workflow timing, foreseeable misuse, and failure conditions. Evaluate performance and usability against those conditions, including cases where the correct behavior is to abstain, request more information, or escalate to a qualified person. Examine whether a safety control actually works in the real workflow rather than simply existing as a button labeled "review."

For clinician-facing decision support, a meaningful review can require access to the basis of the recommendation, training to interpret it, authority to disagree, time to act, and a workflow that does not pressure the clinician to rubber-stamp. FDA notes that patient or caregiver functions do not meet the statutory clinician-limited non-device CDS criterion, and that some time-critical or directive clinical functions remain device software functions. FDA CDS policy navigator This is why adding a clinician confirmation screen does not by itself settle regulatory status or safety.

Monitor, stop, repair, and learn after release

Every release needs a named safety owner, a monitored deployment plan, an ability to disable the affected function, and a way to recover from a bad change. For an administrative system, this may mean feature flags, migration rollback or forward-repair, access revocation, queue replay, and patient-record correction workflows. For clinical functions, it also means a documented pathway to identify affected users or patients, notify the appropriate organization, preserve evidence, investigate the cause, and make a governed remediation decision.

Monitor both technical and safety-relevant signals: authorization failures, unusual record access, mismatched patient identifiers, error and latency rates, tool or model version, output abstention, override patterns, clinician complaints, data-quality changes, bias indicators where valid measurement exists, and near misses. Keep an immutable release record linking the exact code, dependencies, configuration, agent task, tests, approvals, and deployment window. Do not log raw patient data merely to make incident diagnosis easier.

Use staged deployment, shadow evaluation when possible, scoped canaries, fixed stop criteria, and a rollback or safe-disable plan. An AI coding agent must not change a production model, clinical content, access policy, or safety threshold autonomously. A successful build and a passing ordinary test suite are necessary technical signals, not patient-safety evidence.

Prepare an incident response path before deployment. It should define who can stop the function, how to preserve logs and artifacts, how to assess patient and privacy impact, when to engage security and clinical leadership, and how to meet applicable reporting or notification obligations. The exact legal obligations vary by jurisdiction and product status. Obtain specialized advice rather than relying on a general AI-development checklist.

Current regulatory boundaries

The following is a high-level map as of 2026-09-01. It is not legal advice, and it does not replace jurisdiction-specific classification, quality, privacy, or reimbursement analysis.

EU guidance on the MDR, IVDR, and AI Act explains that an Article 6(1) AI Act high-risk classification does not itself raise a device's MDR or IVDR class. MDCG 2025-6 FAQ The point for engineering is not to treat this as a paperwork issue. The intended purpose, device classification, AI functionality, clinical evidence, change control, and post-market process need to be designed together.

A practical adoption sequence

1. Write the intended-use statement and non-use statement for each feature before requesting generated implementation.
2. Map data flows, users, trust boundaries, clinical impact, failure modes, and foreseeable misuse. Assign a preliminary risk tier.
3. Decide which agent tasks are permitted for that tier. Start with documentation, synthetic tests, constrained UI, or non-clinical scaffolding.
4. Establish sandboxing, least privilege, provider approval, data minimization, secret controls, dependency controls, and audit logging.
5. Build the requirement-to-evidence traceability matrix and define independent review gates before code is merged.
6. Run automated security and quality checks, then obtain the appropriate engineering, clinical, safety, privacy, and quality approvals.
7. Release in a controlled environment with monitoring, stop criteria, rollback or safe-disable capability, and incident ownership.
8. Reassess when intended use, data source, user, model, clinical claim, integration, or deployment setting changes. A small wording change can be a material risk change.