Why do relatively few machine-learning candidates have practical AI-security or adversarial-testing skills, and how can they acquire them?

First, separate the three kinds of security work

“AI security” is too broad to be a useful interview requirement unless the system and the expected depth are named. A production AI feature is still software that handles identities, data, networks, APIs, dependencies, and logs. It therefore needs ordinary application security. It also has risks arising from learned models and data, and, if it uses generative models or tools, risks from untrusted instructions and model outputs.

NIST makes this point directly in its GenAI secure-development profile: AI systems inherit risks from ordinary digital systems, while model code, data, weights, and user queries add AI-specific concerns. The profile is intended to supplement, not replace, the Secure Software Development Framework. NIST SP 800-218A

The table is not three separate checklists. It is a reminder that a chatbot with retrieval and a ticket-creation tool can fail through ordinary broken access control, a poisoned knowledge-base document, or an indirect prompt injection. A model-only test will miss much of the risk.

NIST's current adversarial-ML taxonomy explicitly covers predictive and generative AI, including poisoning, evasion, privacy, and misuse attacks across the AI lifecycle. NIST AI 100-2e2025 MITRE ATLAS is a complementary living knowledge base of tactics, techniques, mitigations, and case studies for AI-enabled systems. MITRE ATLAS

Why the combined skill set is often scarce

There is no good public statistic that says what percentage of ML candidates can perform practical AI-security testing. The observed sample does not answer that question. Still, several structural reasons make the gap unsurprising.

The disciplines have different foundations and incentives

ML education usually rewards modelling, statistics, data engineering, and predictive quality. Application-security education emphasises attack surfaces, identities, secure design, code review, incident response, and adversarial thinking. GenAI application security adds model behaviour, retrieval, tool orchestration, and non-deterministic outputs. Each requires substantial practice, and employers often write roles around only one of them.

That creates an ownership gap: an ML engineer may assume the platform team owns security, while the security team may not know which documents are retrieved, how prompts are assembled, or which tools an agent can invoke. This is an inference from the technical scope, not a claim that every team has the problem.

The deployed system changes faster than a course syllabus

The technical guidance is maturing rapidly. NIST finalized the updated adversarial-ML taxonomy in March 2025, and OWASP's current 2025 list treats prompt injection, sensitive-information disclosure, data and model poisoning, improper output handling, excessive agency, system-prompt leakage, and vector or embedding weaknesses as distinct GenAI risks. OWASP Top 10 for LLM and GenAI Applications Universities and general ML programmes cannot turn every new risk category into a mature lab course immediately.

Practical testing needs an authorized target and a clear scope

Useful adversarial testing is not merely asking a public chatbot provocative questions. It requires a defined system, permitted test environment, threat model, expected controls, logging, and a way to decide whether a result is a real security finding. Production deployments may contain confidential data and vendor restrictions, so candidates rarely get permission to test deeply unless an employer creates that opportunity.

Specialists need different depth from builders

It is reasonable to expect a person deploying an AI product to recognise basic abuse cases and escalate them. It is usually unreasonable to expect every ML candidate to be a specialist in adversarial research, penetration testing, cloud security, privacy engineering, and secure agent architecture. Baseline awareness should be broad; specialist depth should be role and risk dependent.

The defensive threat map a candidate should know

The goal is not to memorise attack jargon. It is to ask, for each system: what is valuable, who could influence it, what can that person cause, and what control independently limits the damage?

These controls use classic security ideas deliberately. The key design rule is: the model may suggest an action, but deterministic code and access-control policy must decide whether it is permitted. Never give an LLM a credential that can do more than the bounded task requires.

What defensive adversarial evaluation actually involves

An evaluation is a repeatable experiment that tests a stated risk. It is not a score obtained from a single prompt, a single model version, or an unrecorded demo. NIST's AI Risk Management Framework organises ongoing work as Govern, Map, Measure, and Manage. Its core says to choose metrics for the most significant mapped risks and document risks that are not or cannot be measured. NIST AI RMF Core

Use this process for a small, authorized system:

1. Govern: name an accountable owner, define the system's intended use, legal and privacy constraints, test environment, and stop conditions.
2. Map: diagram data flows, trust boundaries, users, tools, model providers, retrieval sources, identities, and high-value assets. Write three to five plausible abuse cases for the system, not generic internet threats.
3. Measure: create a test set with expected safe outcomes. Include normal tasks, benign edge cases, and controlled adversarial cases. Record the model and application versions, configuration, test date, inputs, outputs, tool-call attempts, and reviewer decision.
4. Manage: fix or reduce the important risks, rerun the same tests, document residual risk, and set monitoring or review triggers for deployment.

For predictive ML, a robustness test might measure whether a defined distribution shift or controlled input perturbation changes a classifier's error rate beyond a pre-agreed tolerance. For a GenAI assistant, it might measure the rate of inappropriate disclosure, unauthorised tool-call attempts, or failure to cite an authorised source in a bounded, synthetic test corpus. The metrics and tolerances depend on consequences. Do not make up a universal pass percentage.

NIST's Dioptra is a security-evaluation test platform designed to measure, analyse, and track AI risks, and its documentation includes a controlled adversarial-ML tutorial for an image classifier. It is a useful reference for reproducible experiment design, not a substitute for a threat model. NIST Dioptra overview Dioptra advanced tutorials

A minimal evaluation record

This record shows the candidate understands that security testing needs a system objective, an observation, and a response plan.

A realistic learning roadmap for candidates

Assumption: you already know basic programming and can build or inspect a small ML or LLM application. If not, begin with programming, HTTP and APIs, Git, basic cloud or deployment concepts, data handling, and the ordinary software-security foundations below.

Stage 1, weeks 1 to 2: learn the secure-software baseline

Learn authentication versus authorization, secrets handling, dependency risk, input validation, logging, least privilege, incident basics, and threat modelling. Read the NIST Secure Software Development Framework alongside NIST SP 800-218A to see which standard practices still apply when the application happens to use a model.

Deliverable: Threat-model a familiar non-AI API. List its assets, entry points, identities, permissions, abuse cases, and evidence you would want before release.

Stage 2, weeks 3 to 4: learn the ML-specific vocabulary and lifecycle

Read the executive summary and taxonomy of NIST AI 100-2e2025. Learn the defensive meaning of evasion, poisoning, privacy attacks, and extraction. Focus on the attacker capability and lifecycle stage required for each risk. That stops you from treating every research result as equally probable in your product.

Deliverable: Add a data-and-model supply-chain section to your threat model. State who can change training, fine-tuning, or retrieval data, who can publish an artifact, and which integrity controls exist.

Stage 3, weeks 5 to 6: learn GenAI system boundaries

Study the OWASP 2025 risks that match your architecture: prompt injection, sensitive disclosure, data or model poisoning, improper output handling, excessive agency, and vector or embedding weaknesses. OWASP Top 10 Map every user input, uploaded file, retrieved document, web page, tool response, and model output as trusted, untrusted, or conditionally trusted. In most systems, treat them as untrusted until an independent control says otherwise.

Deliverable: Draw a one-page data-flow diagram with trust boundaries. Mark where authorization is enforced, where secrets live, and where a human must approve an action.

Stage 4, weeks 7 to 8: build a safe evaluation harness

Create a small offline or sandboxed system using public or synthetic data. Write a labelled test set before changing defences. Include ordinary user tasks and safe representations of the risk cases you have mapped. Test only systems you own or are explicitly authorised to assess. Do not test public services, production systems, or third-party data without written permission.

Deliverable: A versioned test corpus, a runner that saves structured results, and an evaluation report that compares the baseline and revised configurations.

Stage 5, weeks 9 to 10: harden, review, and explain

Apply controls that reduce the particular risk: per-resource authorization before retrieval, strict output schemas, server-side policy checks, limited tool scopes, confirmation for impactful actions, data review, and logging. Then rerun the identical suite. Ask a security practitioner or experienced engineer to review the diagram and report.

Deliverable: A concise case study with the unresolved risks and proposed next test. A good security report is honest about what it did not test.

Keep learning without tool chasing

Follow the release notes and primary guidance for the systems you use, then revisit the threat model when you add a new model, tool, retrieval source, multimodal input, permission, or deployment environment. MITRE ATLAS can help turn observed techniques into new test hypotheses. MITRE ATLAS

Portfolio project: a defensible policy-assistant sandbox

Build a fictional internal policy assistant using only public or synthetic policy documents. The project should answer policy questions, cite the source snippets it used, and have a narrow, read-only lookup function. Do not use real employee records, student data, customer data, secrets, or live integrations.

Project design

1. Start deliberately limited. Use a small fixed document corpus and no write-capable tools. A user identity can be a mock role, such as engineering or hr, rather than a real account.
2. Enforce access outside the model. Filter documents using server-side role and document metadata before retrieval. The model must not decide whether the requester is allowed to see a document.
3. Keep untrusted content separate. Label retrieved text as untrusted reference material. Do not allow it to define access policy, tool permission, or system instructions.
4. Validate output before display or action. Render plain text safely, require source citations from the allowlisted corpus, and pass any structured output through a strict schema validator. Do not execute generated code, queries, or commands.
5. Instrument the system. Log sanitized request metadata, retrieval IDs, attempted tool calls, policy decisions, model and prompt version identifiers, and reviewer labels. Do not log confidential content or credentials.

Safe test categories

Use a test sheet with these categories, written at a harmless, abstract level:

Do not publish exploit strings, live credentials, or instructions for bypassing another service. The educational proof is the threat model, controlled test design, control implementation, and before-and-after findings.

What to publish in the portfolio

• A one-page architecture and trust-boundary diagram.
• A short threat model that names assets, actors, assumptions, and excluded risks.
• A safe, synthetic test manifest with expected outcomes.
• A reproducible evaluation report. Include results as counts and labelled failures, not impressive-looking but unsupported percentages.
• A control matrix that says which risks are reduced, what is still possible, and when a human review is required.
• A three-minute demo in which you explain why authorization and tool policy live outside the model.

Hiring assessment example: test judgment, not stunt-prompt skill

For an ML engineer who will deploy a RAG or agent feature, use a 60-minute, take-home or live design exercise in a disposable environment. Give the candidate a simple architecture: a web client, an API, an LLM, a document store containing department-labelled documents, and a ticket-creation tool. State that the assistant should answer internal-policy questions and that it must not create tickets without approval.

Ask the candidate to produce a short design note, not to attack a real system:

1. Identify assets, actors, trust boundaries, and three high-priority abuse cases.
2. Distinguish what ordinary AppSec, ML security, and GenAI security each contribute.
3. Propose a control design for document authorization, untrusted retrieved content, tool permissions, and model output handling.
4. Describe a small, authorized test suite and the measurements that would make them pause a release.
5. State what they would ask an AppSec or AI-security specialist to review.

This assessment is fairer than requiring a candidate to invent unrestricted jailbreaks. It measures secure-design reasoning and communication, and it does not reward unsafe behaviour.

Hiring decision rule

For a normal product role, hire for strong engineering or ML fundamentals plus demonstrable security judgment, then provide internal secure-AI training, review practices, and specialist access. For a role that owns adversarial evaluation, model-security research, sensitive data, high-impact agents, or regulated deployment, require deeper specialist evidence and give the role time and authority to test properly. Do not call a product secure merely because an ML engineer passed an interview. Security is an organisational capability.

Practical release checklist

Before a team ships an AI feature, it should be able to answer these questions:

• Is there a current architecture and data-flow diagram with trust boundaries?
• Are user identity, authorization, and tool permissions enforced by code and policy outside the model?
• Are training, fine-tuning, retrieval, and model artifacts sourced, versioned, reviewed, and protected according to their risk?
• Is all model output treated as untrusted before it reaches a browser, database, API, or other system?
• Has the team tested direct and indirect prompt-injection scenarios relevant to its own inputs and retrieval sources?
• Are tool capabilities minimal, reversible where practical, logged, rate-limited, and subject to approval when consequential?
• Are tests repeatable, versioned, and rerun after meaningful model, prompt, data, tool, or access-control changes?
• Is there an owner for residual risk, incidents, user reports, and escalation to security, privacy, legal, or compliance teams?

If several answers are “no,” the responsible next step is to reduce scope, remove the risky integration, or delay release until the controls and review exist. NIST's AI RMF does not promise a universal checklist, but it supports this iterative approach of mapping, measuring, managing, and documenting risk. NIST AI RMF Playbook

Limits and safety boundaries

Security guidance cannot guarantee a system is safe. The attack landscape, model versions, integrations, permissions, and data change. Some robustness or privacy risks are difficult to measure, and a passing test suite proves only what it covered.

Use only systems and data you own or are explicitly authorised to test. Keep tests in isolated, least-privilege environments, use synthetic data whenever possible, and coordinate findings through the organisation's security process or a vendor's authorised vulnerability-disclosure programme. Do not access, extract, alter, or disclose another party's systems, data, models, accounts, or secrets.

AI features handling health information, financial decisions, legal matters, education records, children, employment, credit, or other high-impact uses require additional privacy, security, legal, and domain review. Applicable requirements differ by jurisdiction and use case. This answer is technical education, not legal, security-certification, or compliance advice.