On Designing (Good) Technical Interviews
I've taken my fair share of technical interviews. And, as I've frequently been put in positions where I help in the hiring process, I've also been tasked with helping design them.
Technical interviews get a bad reputation, and for good reason! They are often based on trivial or irrelevant problems which cast the candidate in a demeaning spectacle of power-imbalance. But, they don't have to be awful; and, when crafted and administered well, they can be beneficial for both the interviewer and the candidate!
Speaking in the Abstract
The interview always consists of “a prompt”—the challenge or task the candidate is tasked with completing. Though I phrase this as if it is singular, it may well be a set or series of tasks; mentally adjust the pluralization or nomenclature as-needed.
For the interview to be most useful, the prompt should…
- … not result in the candidate performing real work without compensation1
- … be realistic enough that the interviewer gets some assurance that the candidate, upon successfully completing the prompt, could do the work required; and, equally as important, a reasonably-realistic prompt also helps the candidate evaluate whether or not they personally feel capable and interested in doing it2
- … for good-fitting candidates, take long enough that it's not trivial-to-complete (either by copy/pasting, or in little more than a couple lines of code), but never so long that it's a slog or otherwise painful to complete3
- … be standardized and repeatable, ideally to the point of enabling the fair comparison of multiple candidates
The Platonic Ideal
Easily the best technical interview I've ever been able to adminsiter was to give the candidate an incredibly short-term contract (e.g., 3 days). The prompt, then, is to work with me, on a real problem, for the term of the contract. This format leans into the realization that (1) is in direct conflict with (2); it reconciles them by giving the candidate real work and real working conditions while paying them for it! This has a lot of benefits:
- The candidate gets to not just feel, but experience both what the actual work and the actual working environment are like
- The candidate's time is completely compensated, so the format doesn't inherently take advantage of them
- The interviewer gets to actually work with the candidate on something real—a much better barometer of real-world performance and working relationship than any other format I've seen
- The interviewer gets to continue making progress on regularly-scheduled work during the interview period (with help if the candidate is doing well!)
But this format is not a total panacea. There are at least a few drawbacks that I can see:
- A total loss of (4). It's very unlikely that a real, useful, and representative task needs to be accomplished multiple times; so, standardizing this format enough for fair comparison seems impractical.4
- It's rather expensive for the hiring organization.5
- It considerably narrows your candidate pool. Not all candidates are going to be able to accept such a situation. Particularly, anyone currently holding a position may be hard-pressed for this format to function.
A Close Second
In lieu of the using the above, I think the best way to strike a balance across the four properties above is for the prompt to be based on real work which the team has previously completed. Because the team is already familiar with the problem, a realistic and representative prompt is easier to craft, and the solution can be measured more fairly against the solutions the team actually implemented. What's more, no real work is being asked of the candidate; and, because it is synthetic, it can be standardized and repeated.
The hard part now is to suitably simplify the prompt so that it is still representative without being awful to perform. It is incredibly easy to underestimate the difficulty of performing some work under pressure—especially when you're already familiar with the particulars. If you have any doubt, make the prompt easier!
Additionally, to give the candidate the best chance possible, it's ideal to make the challenge low-stress. In my experience, this is best-accomplished with a takehome format—the candidate is not put on-the-spot, and can complete the prompt in the development environment with which they are most comfortable. Takehomes are contentious, but in this case, it is done explicitly out of the hope that it makes the process smoother and more humane. Additionally, the takehome should never be the final phase of the interview! Barring incredibly extenuating circumstances, all candidates who submit a solution should benefit from a technical debrief with the interviewer after an internal review of the submission.
An Aside on Timing
To try to minimize pain for the interviewee, the prompt should be crafted with a particular amount of time you expect good-fitting candidates to expend. I try to target no more than 1.5 hours, and I communicate to candidates that they should spend no more than 4 hours total.
You cannot necessarily control how much time candidates actually spend on the task, but you can bound it. I tend to give the candidate the freedom to complete the task when they would like to, but require submission within a week of the task being assigned. Because the interviewer will debrief with the candidate after-submission, you can also make clear to the candidate that even if the solution is not complete within the time limit, that is not necessarily a “fail.” I have interviewed people who submitted partial solutions and gave a “yes” referring to the hiring manager; at least one of those candidates went on to be hired, and they were a good addition to the team!
Making it clear that solutions are not pass/fail, along with all the work that has gone into making this process as pleasant as possible hopefully decreases any incentive the candidate might have to “cheat.”
The Myth of “One Size Fits All”
If it isn't already clear, this format requires that the specific prompt must be tailored to the intended position (or, at the very least, the team) in the organization. The best prompts are modular; this allows the same tasks to be tweaked to function well for multiple positions, or to enable configurable difficulty for the intended level of experience or seniority.
This does mean this format is specifically not well-suited to evaluating a candidate for general, technical fit without an intended position. In particularly large organizations, that may be a non-starter.
Below are a few high-level prompt descriptions I've seen work well in the past:6
-
For a Database Administrator:
- Here's a set of data we have in an excel spreadsheet for X purpose. Design a simple schema to hold this data in a database that will achieve reasonable performance for our purpose. Talk about tradeoffs in the design you chose.
- Here's a simple database schema we have made, and some example data from its tables. Query X and Y are performing terribly; what would you do to improve the performance of both?
-
For a Full-stack Web Developer:
- Here's a simple database (or filled, static data-structure). Make a simple website to enable feature X atop this database. Talk about how you built it, and what led you to the choices you made.
- Here's a simple website we built, and an issue tracker that includes some tickets. Address and close three tickets; include rationale and implementation notes in the issues as you close them.
-
For a Distributed-systems Engineer:
- Here's a simple distributed system which exhibits unfortunate behavior X. Fix it. Talk about alternative solutions, and why you ruled them out.
The Interview Lifecycle
With this format as the core technical interview, the candidate's interview phases can be contracted significantly from many-phase interview processes common in other formats:
- An initial screening call to ensure the candidate is a real, human person who actually wants a job and vet they have aligned interests and principles for the organization. (approx. 30 minutes)
- The takehome challenge with a flexible schedule to complete per their availability, as mentioned above. (cf. An Aside on Timing)
- (An internal review of the takehome, ideally by multiple people.)
- A live debrief and review of the takehome with the candidate and reviewers. This provides an opportunity for the candidate to reflect, and ask for feedback and technical clarifications on the task.7 (No more than 1.5 hours, probably closer to an hour)
- A final interview with the hiring manager or the person with the final decision-making authority (approx. 30 minutes)
This works out to no more than 2.5 hours of interviews for the candidate (plus hopefully no more than 4 hours of technical work), where each interview is unitary and serves a specific, clear purpose.
“Patches Welcome!”
I still don't think this format is perfect, but it's the best I've found so far. If you have ideas for how this scheme could be improved, reach out! Hiring is a very strange balancing act, and I always want to be making the process better for everyone.