If a candidate uses an AI agent in your interview, the code they hand in tells you less than it used to. What tells you more is how they got there.
So stop grading the final code on its own and watch how the candidate directs the agent: how they break the problem down before the first prompt, how precisely they say what they want, whether they check what comes back, and how they recover when it goes wrong. Then ask them to walk you through what got built, because a candidate who cannot explain the result did not really direct the work.
The rest of this post is how to do that in practice, with a rubric at the end you can copy into your own interview loop. It works with or without any tool.
What should you watch when a candidate works with an agent?
Watch the session as a sequence of decisions, not as a race to a green test suite. The useful moments happen in roughly this order.
Before the first prompt. Do they read the problem, name the pieces, and decide what to hand off and what to do themselves? Or do they paste the whole ticket into the agent in the first minute? This is decomposition, and it is the behavior that separates someone directing an agent from someone watching one.
The ask. Is the instruction specific about inputs, constraints and what "done" means? "Add pagination to the list endpoint, cursor based, keep the existing response shape, and add a test for the empty page" is a different skill from "make the list endpoint paginated". Call it spec precision.
While the agent works. Do they run the tests, read the diff, and try the edge case the agent skipped? Or do they accept the first output that compiles? Verification is where most of the real engineering judgment shows up, and it is invisible in the final code.
When it goes wrong. Every session has a moment where the agent goes off course. Good candidates diagnose it and redirect with new information. Weaker ones re-run the same prompt and hope for a different answer. Recovery is one of the most telling things you can watch.
At the end. Does the result do what the problem asked, or what the agent drifted into? Agents are good at producing something plausible. A candidate with strong intent fidelity notices when plausible is not the same as correct.
On the developer side, kodwai uses these exact labels for its signals: Decomposition, Spec Precision, Verification Rigor, Recovery and Intent Fidelity. You do not need our vocabulary. You need to watch for the behaviors.
How do you judge decomposition before the agent touches code?
Give a problem that has to be split. A single function is too small: the candidate can hand it over whole and you learn nothing. Something with more than one file and at least one real constraint (a performance budget, an existing interface that cannot change, a data shape that is awkward) forces a plan.
Then watch the first few minutes closely. Two ways to get the plan out of the candidate:
- Ask them to say their plan out loud before they prompt anything. Keep it short and informal. You are not grading a design doc.
- Or say nothing and read the plan from their first prompts afterwards. A candidate who decomposes well leaves a trail: the first prompt sets up structure, the next ones fill it in, and each one is scoped.
What weak looks like is also easy to spot. The whole problem goes in, a wall of code comes out, and the rest of the session is spent patching whatever broke. That can still end in passing tests. It just tells you the agent did the thinking.
Why "explain what got built" still matters
After the session, spend a few minutes on a walkthrough. Why is it structured this way? What would break first under load? What would you test next if you had another hour?
This is the fastest way to separate directing from delegating. A candidate who steered the work can answer from memory, because they made the decisions. A candidate who delegated everything will describe the code the way you would describe someone else's pull request.
Pair the walkthrough with the session record if you have one, so the conversation is about what actually happened rather than a rehearsed answer. "You rejected the agent's first approach here, why?" is a far better question than "tell me about your design."
Is the final result still worth scoring?
Yes, as one input. The tests should pass and the code should be something your team could maintain. That has not changed.
What changed is that a clean result no longer proves the candidate understood it. Passing tests is necessary but not sufficient. Score the outcome, then weigh it against everything you saw getting there.
Short observed session or paid work trial?
Some teams are moving to paid work trials: a few days of real work, paid, on something close to the actual job. It is the closest signal to the real thing. It also asks a lot of the candidate, who may already have a job, and a lot of your team, who have to set it up and review it.
A short observed session is cheaper to run and much easier to compare across candidates, because everyone gets the same problem under the same limits. It shows less than a trial does.
These are not mutually exclusive. One approach that avoids the worst of both: use the short observed session as the filter, and save the paid trial for finalists, where the extra cost on both sides is worth it.
A rubric you can use without any tool
Copy this into whatever you already use for interview feedback. Score each row however your team scores today. The point is to make interviewers look at the same things.
| Behavior | Strong | Weak |
|---|---|---|
| Decomposition | Names the pieces before prompting, decides what to hand off, scopes each ask | Pastes the whole problem in the first minute |
| Spec precision | States inputs, constraints and what done means | Restates the ticket and lets the agent guess |
| Verification | Runs tests, reads diffs, tries the edge case the agent skipped | Accepts the first output that compiles |
| Recovery | Diagnoses the failure and redirects with new information | Re-runs the same prompt, or starts over from scratch |
| Intent fidelity | Result does what the problem asked, notices drift | Result does something plausible but different |
| Explains the result | Walks through decisions and trade-offs from memory | Describes the code as if reading it for the first time |
A candidate can be strong on the outcome and weak on every row above. That is exactly the case this rubric is meant to catch.
Where kodwai fits
We built the kodwai hiring track to make this kind of interview easier to run. Teams create a custom interview project and send it with a one-link email invite. Candidates solve it locally in Claude Code, on their own machine. Interviewers can watch the session live: the prompts, commits, test runs, tools used, time and API cost.
After the session, kodwai scores the candidate against your own per-role rubric, each dimension with a written justification, strengths and weaknesses. Your team adds its own scores and comments, and you can compare candidates side by side. Roles (admin, interviewer, viewer) and time, budget and tool limits are set per interview.
What it does not do yet: there are no ATS integrations, no published pricing and no video playback of the session. Not yet, and we will not pretend otherwise.
If you want to try it on your next role, the details are on the kodwai hiring track. Reach out and we will get your team set up.