How should teams interview software engineers when candidates use AI coding tools?

Decide what the role requires

Start with a short analysis of the job, not a preference about AI. List the work behaviors a successful engineer must perform in your environment: understand an unfamiliar service, turn a requirement into a safe change, inspect a diff, identify risks, write or revise tests, debug a failure, use internal documentation, communicate uncertainty, and know when to ask for help. Then decide which of those behaviors normally involve an AI assistant.

That step keeps the interview tied to the role. The APA and SIOP selection principles say a selection procedure should be based on analysis of the work and should distinguish capabilities expected before hire from those the employer will train. APA and SIOP personnel-selection principles. A work sample is a job-specific test that replicates day-to-day tasks, which is why a small debugging or review task is usually more informative here than a puzzle disconnected from the team’s software. APA Dictionary definition of a work-sample test.

For a role where engineers routinely use an approved assistant to investigate, draft, or review changes, the assessment can include that assistant. The skill being assessed is broader than prompt writing. The candidate needs to check assumptions, identify what the tool cannot know, validate code against the system’s rules, and take responsibility for the final change. If the job instead requires maintaining a restricted environment where such tools are unavailable, assess that constraint directly rather than presenting an AI ban as a generic test of engineering ability.

Give candidates a realistic exercise with declared rules

Choose one small scenario from the team’s actual work, but build a self-contained replica with synthetic data and no confidential source code. Provide a compact repository or code excerpt, a failing test or incident report, acceptance criteria, and the relevant documentation. Include one meaningful constraint such as backward compatibility, authorization, latency, cost, or an external API retry. Tell the candidate whether the objective is to propose a patch, review a supplied patch, diagnose a defect, or decide what information is missing.

State the AI rules in the invitation and repeat them at the start. A useful policy answers these questions:

• Which AI tools, local tools, documentation sites, and internet access are allowed.
• Whether the candidate must use a supplied account or may use a personal account. Do not require a paid subscription or assume candidates can safely disclose personal credentials.
• What may be sent to a tool. Use only the provided synthetic materials and prohibit copying private employer code or personal data into the exercise.
• What the candidate should disclose. A short note naming the tool and describing how they used its output is usually enough. Do not score the quality of a private prompt unless prompt design is itself a job requirement.
• How the candidate can request an accommodation or report a tool-access problem without losing time or being penalized.

Use the same rule set for candidates compared for the same opening. Changing tool access halfway through an interview creates an avoidable difference in the assessment. If you want to understand unaided reasoning as well as AI-assisted work, use two clearly labelled parts with separate scores: a brief explanation or review without AI, followed by a normal work sample with AI allowed. Do not silently make the unaided part the real pass or fail test if the role expects AI use.

A worked exercise

Setup. A hypothetical team operates a subscription service. The candidate receives a small service that creates invoices after a webhook retry, a test that sometimes produces two invoices, and a brief API contract. The candidate may use the approved AI assistant and the supplied documentation. The task is to explain the defect, propose a safe change, and add or describe the tests needed before release.

What to observe. A strong candidate may ask whether the external event has a stable identifier, inspect where the retry occurs, and use the assistant to summarize the flow or suggest tests. They then notice that the service lacks idempotency, meaning a repeated request can cause a second external effect. They propose storing or checking a durable event key before invoice creation, discuss a database constraint or transaction boundary, test a duplicate delivery, and explain how a partial failure would be handled. They should also point out any assumption that needs confirmation from the API contract or a teammate.

What it shows. The exercise reveals whether the person can turn an AI suggestion into a reviewed, testable change. A candidate who uses no AI but follows that reasoning can score equally well. A candidate who produces a polished patch without noticing duplicate effects, missing tests, or uncertainty has not demonstrated the production judgment the team needs.

NIST’s current draft update to the Secure Software Development Framework describes code review and code analysis as activities selected by an organization’s policy and software stage, and calls for recorded triage of discovered issues and remediations. NIST SP 800-218r1 initial public draft, pp. 39-40. Use that model in the exercise by observing review, testing, and issue handling rather than rewarding unexamined generated code.

Ask questions that expose reasoning and review

The exercise should leave room for a conversation. Ask the same core prompts of every candidate, while allowing relevant follow-up questions that clarify their answer. Useful prompts include:

• “What would you check before trusting this suggestion?”
• “Which failure path is most likely to reach production?”
• “What test would give you confidence that the fix works, and what would it miss?”
• “What information do you need before you can make this change safely?”
• “If the assistant and the existing test disagree, how would you investigate?”
• “What would make you stop and involve a reviewer, security engineer, or product owner?”

These questions examine the candidate’s model of the system. They also reveal whether the candidate can distinguish a plausible answer from verified behavior. Asking for tradeoffs matters because many production decisions have competing goals: a strict timeout might protect capacity but lose a slow customer request, while a retry can increase reliability but create duplicate effects unless the operation is idempotent.

Include a review task as well as a build task. Give the candidate a small diff, possibly one drafted with AI, that contains a plausible error such as an authorization check after a data fetch, an unbounded retry, a missing null case, or a test that only covers the happy path. Ask what they would change and what evidence they would seek. This reduces the advantage of a candidate who can generate code quickly but cannot inspect it critically.

Score demonstrated engineering skills

Agree on the dimensions and scoring anchors before interviews begin. Have interviewers score independently before discussion, write a short evidence note, and compare a few practice evaluations before interviewing real candidates. The compact rubric below is a starting point, not a validated hiring instrument. Tailor its weights and examples to the job analysis, then review whether it predicts the performance you care about.

The table deliberately gives no points for using an AI tool. AI use can supply evidence for review and communication, but the score should come from the candidate’s decisions and explanations. If the role explicitly includes building prompts, configuring agents, or evaluating model output, add a separate job-related dimension with an observable anchor. Do not let an interviewer reward familiarity with a favorite vendor when that familiarity is not required by the job.

Structured scoring is also a fairness control. The U.S. Uniform Guidelines discuss adverse impact across the total selection process and expect validity evidence for a selection procedure that has adverse impact and remains in use. EEOC questions and answers on the Uniform Guidelines. This does not turn the example rubric into legal advice or a complete compliance program. It does mean leaders should retain the job analysis, exercise rules, scores, and periodic outcome checks rather than relying on interviewer intuition.

Keep the process fair as tools and roles change

Give candidates the instructions, time limit, permitted resources, expected deliverable, and evaluation criteria in writing. Train interviewers on the rubric and on what follow-up questions are appropriate. Use equivalent variants of a scenario when needed to prevent reuse, and check that the variants require the same skills and have similar difficulty. If an AI tool fails, document the failure and offer the same remedy to candidates in the same situation, such as restarting with the time restored or using an approved fallback.

Accessibility needs belong in the process design, not as an awkward exception during the interview. For U.S. employers covered by the ADA, the EEOC explains that algorithmic or AI tools can screen out qualified people with disabilities and that reasonable accommodation may be required in the hiring process. EEOC guidance on visual disabilities, software, and AI. Provide an accommodation contact and make sure the exercise and allowed tooling work with assistive technology. Obtain local employment advice for other jurisdictions and for any high-volume or automated assessment process.

Review the interview process after hiring. Compare rubric patterns among interviewers, candidate feedback, completion rates, accommodation requests, and later job evidence that is appropriate and lawfully collected. If reviewers repeatedly disagree on a dimension, the anchor is too vague or the exercise does not expose the behavior. If the exercise no longer resembles the job because the team changed its tool policy, update the task and tell future candidates. The purpose is a defensible, useful assessment of the current role, not a permanent test of one generation of AI tools.