Ask HN: How do you interview devs in a post-AI world?

Ive been trialing different types of interviews for over 2 years now and here’s what I’ve found.

I tried 2 flavors of a technical interview and this round was 1 hour in total.

The first was a traditional interview problem where we give the candidate a codebase and a docker image to run against. The problem itself involves using the docker image to post and get responses from but the candidate is expected to write a new helper/service in any language they choose. We ask for no agentic workflows for this but any other resource is fair game. This has been the most successful for us to find candidates and generally provides the best experience.

Our full agentic interview involves an existing fullstack typescript codebase that we give to the candidate a week or two before. The candidate is free to use any agentic workflows they want during the interview and we give them a ticket that we went to implement during the interview. I find a ton of variability in this interview and a lot of people will just take the entire ticket and one shot into Claude without prepping. This is lead to the biggest disparity in results of candidates so we stopped doing it.

The same way we did before. Very very simple code submission (most people using AI use it even though we call out we're going to ask them later to modify later without AI tools writing code for them) then pairing interview where we ask some basic "are you actually at the level you say you are" question, then ask them to extend their program submission with:

- engineers on the call as pairing assistants - google, ai tools, whatever for libraries, syntax, etc. we tell the candidate directly that it's impossible for us to gsther signal on how they think about problems if they ask Claude to just whip them up a solution - the expected output - their own unit test suite

Every candidate that has submitted an ai submission thus far has failed because they have literally no idea where to go. I've interviewed dozens at this point. I'm not saying "they're unfamiliar with the structure", I'm saying "they cannot actually break down the problem even verbally".

It doesn't matter their pedigree or past experience on their resume, if they used AI to generate they don't seem to be able to resurrect the skills that actually matter for the thing, engineering and product work.

Note because I know folks hate code submissions. It's not hard. We give a CSV with 3 columns, 10 lines. Do some basic mapping and some structuring, some basic data modeling. We only expect about 1 actual class or struct. Then unit tests and it should run in the terminal. Max submission length with verbosity has been a java program at something like a hundred lines total if that, most folks complete the submission in an hour or two. Extension is that we modify one of the rules and extend the CSV by 5 lines.

I seem to still be getting good signal from this, since the engineers that I've hired off of this have been fantastic with or without AI tooling immediately in their hands during the day.

Prompts, prompts, prompts.

Give them a time-constrained challenging problem that is wide in scope and see how well a candidate can decompose it before feeding it to AI. You can get pretty high signal within the first 2 prompts. Are they able to effectively steer the model or do they just ride with the flow and accept every AI suggestion?

How well do they know their models and their limitations? Are they able to switch tools/models effectively on the fly depending on the task? Or are they just using cursor auto mode and copy-pasting the task description into their IDE? Do they have a custom harness/workflow? What skills are they using, if any?

Judge them on quality of the output first. And pay attention to their taste.

To me AI hasn't had any impact on interviewing, only on filtering the CVs as I immediately flag obviously LLM-generated ones that don't have any substance, while those that are LLM-assisted with tasteful judgement are fine.

On the interview side, I have gone through a phase which required do an architectural design task on a whiteboard. I realized how stressful and not very helpful at evaluating it might be when I myself had to sit on the interviewee's side several times. It's difficult to get into this play-design mode.

Currently, I just make natural conversations, ask some technical questions on the level where I estimate the candidate is, try to understand the level of their knowledge and whether I would like to work with them in one team as a person. Maybe it wouldn't work for highly specialized roles, but for a general backend work I have been happy with the results.

Measure them on merit, their body of work and how well they will fit into the team culture.

As for the questions you’re asking, instead of asking old-school coding interview questions ask them stuff that is actually relevant to agentic programming.

Ask them to build a harness, have them explain what skills are, have them build a self-made implementation of Claude Code as take-home assignment.

The times have changed, you need to update interviewing questions to meet them. Start with agentic questions and ramp them up into more complex scenarios. This is how you gauge your candidates, see how they think and solve their way out of it.

Try to determine how creative, adaptable, motivated the candidate is. Figure out what there actual skill-sets are that they can bring to you, not whether they know some CS pedantry

Leetcode doesn’t tell you anything apart from someone memorised set of questions and their solution.

For your question, I assume you want people who can solve problems, can explain their thought process and reasons of decisions. I did interview devs recently, and my questions were related to their experience in their CV. Something like, tell me about project X in Y company, what did you do, what did you use, why, what did you learn from it.

My personal opinion is that LLMs are no difference from developers who over engineer and over complicate things. You can get them to do good work with the right steering and right understanding of the whole system within the context of the company.

i think it would make more sense to let developers whatever coding agent they want to use to build something complex and then have them explain every bits and pieces of what was built, why, alternatives, design, scalability considerations etc.. this would result in much better technical discussion instead of having someone code manually. every developer needs to use a coding assistant today to build what they need to with the best quality in short time. so interviews need to be way to see if the developer can do that in real work.

I am very curious about this as well, as an IC. I have not written code by hand for almost two years now and have produced a mountain of Python code over the past year (well architected, using all of my experience in other languages to support good judgement). However, Python is new to me -- I've never hand written a line.

This is all fine and good at my current job, but were I to ever go back on the market. What an uncomfortable conversation!

How does one approach this weird state we're in?

Ask them how many "R"s are in the word "strawberry"

Recently, I was screen-interviewed for coding speed with a fairly easy project. The company used a 3rd party platform which gave me 90m to complete 4 levels, each one building upon the other.

I think that was a nice test, but the platform is too buggy to recommend.

Conventional technical interview (DSA, whiteboard architecture).

Then 1-2 day paid work trial.

Interestingly…no one has said no to the trial.

It turns out that people want to derisk their decision just as much, if not more.