How to make coding Interviews better
My lessons from 100s of interviews being on both sides of the table.
I’ve done over >100 interviews in last 5 years. I am not an expert. However from my experience, I have some theories on interviewing and some recommendations on how we can fix Software Engineering interviews.
On the other hand, I have also interviewed at other companies >50 times (mostly to gauge what my worth in the market is) so I can speak to both sides of the table.
Problem
An interview panel usually consists of 3-5 different interviews. A phone screen, an interview with hiring manager, and a few other technical interviews. Per candidate, a company spends about ~8 hours. The goal is to find someone with the relevant skills who can fill a position and deliver more value than their compensation in a few years.
An interview with an interviewer usually lasts 45m-1.5hrs, depending on the format. One of my ex-managers (Alex) used to say “you have to decide whether this person is worth a Lamborghini, because the company will spend Lamborghini money on his/her compensation every year at 200-300k/year (including office space + benefits)”.
I’ve also asked myself, what correlates with good long term performance. Did my bets on candidates pan out?
Disclaimer
The data is very sparse. Very hard to predict how a person will perform for multiple years based on an hour of interview. Even if it’s a panel, they all only get an hour’s view of the person. Many times, it’s not even the person, the org fails to integrate and mentor them them so they can do their best. So if you fail an interview, realize it’s a number’s game and don’t take it personally. We’re all judging books by their covers.
The industry at large hires only when they have a strong signal. When I’ve failed an interview the #1 response is “We didn’t get a strong signal”.
What has worked well
The biggest signal in my mind (even more so than code) is 3 things.
Being a nice human being
Experience of a candidate in a problem space
Their ability to iteratively overcome obstacles.
A specific resume - I try to read resumes twice before an interview. On the other hand I loathe when an interviewer hasn’t read my resume and takes my time for granted. Abstract words like “hard working, results oriented e.t.c”, mean very little. To show a strong signal, mention specifics: “I did X part in building Y, that had Z outcome”.
If candidate has linked their github username in resume, I look around their github profile to see what they’ve built. Even if they don’t do the best on coding part, some evidence of them writing decent code and collaborating with others to get things done is a strong signal.
Referrals - Birds of a feather flock together, the word of someone you trust carries a lot of weight. Especially personal referrals i.e “Have you personally worked with this person and vouch for their competence?” In a similar manner, it’s important to call references and see what they have to say. Ask specifics from references “what did they do? why would you work with them again?” Talent attracts talent. I’ve also found value going on the Candidate’s LinkedIn to see if there’s anyone in my network that can vouch for their experience.
^ are signals you can get without any face time with the interviewee. Below are my learnings on conducting interviews.
The coding interview
In my experience, I see little correlation between getting the correct answer and performing well. I have however noticed correlation between how a candidate arrives to a solution.
Interview example: Write a function that converts an integer e.g 100 to “one hundred”. I’ve found little correlation between trick coding questions involving complex data structures and long term performance. Usually keeping it to the basic works.
Signal #1: Clarify problem-space
Does the candidate probe about the boundaries of the problem? How big can the numbers be, can they be negative? If it’s a dynamic language “Are they always numbers? what about decimals and fractions?”
Most real world problems are ambiguous, digging into the fundamental problem and clarifying scope is essential.
Signal #2: Debate viable approaches and align on rough plan
Rather than start writing code directly, a good candidate would explain their approach and mental model. Doing this early with interviewer means they collaboratively figure out any holes in approach before writing actual code. At this stage, using a whiteboard can be effective. Using a notebook can be effective too. Waving hands is cool too.
Most real world problems require alignment between many people. Being able to communicate what’s in your mind to others and articulate pros/cons of different approaches is essential.
There is no one right answer answer to most large problems, they come with tradeoffs. Aligning on tradeoffs is a good signal.
Signal #3: Natural environment
Writing code on a whiteboard is really awful. It has little correlation to how day to day work gets done. Writing code on coderpad is painful too. There is no code completion, no auto-formatting. Most engineers I know have their IDE setup: vscode, sublime, vim e.t.c
I’ve found candidates perform a lot better when you let them use their natural environment. Sure it’s an interview environment and they still have a bit of time pressure, but it’s closer to real world work.
I did an A/B test in giving the same question letting candidates use their laptop and screenshare, using coderpad.io, and sandbox.io. Candidates performed better when using sandbox.io for frontend questions since there is less friction to get started.
Candidates using their own laptop and sharing over zoom by cloning a repo was also a strong signal. There’s only so much you can do in coderpad. By cloning a repo you can get a signal on how a candidate is able to jump into an unfamiliar codebase and navigate around it.
On a Google interview, they made me write code in google docs. It was not fun. In my experience, Stripe does interviewing well. You checkout github repos and work on your personal laptop with your fav IDE.
Signal #4: Code -> Run -> Debug iterations
Write something small, run it and see if it produces the right answer . Candidates that iterate and catch silly mistakes early do better. I’ve seen candidates that keep on writing code and never run until their interview is over. When they run, the code doesn’t compile or produces an error.
Table tests are also a strong signal. Having an array of [[input, output], [input, output],…] and passing it to a simple harness function is great. Seeing test cases and code complexity evolve is a delight.
Summary
It’s essential to test a candidate with a coding question and an environment that is close to real world work. Give more weight to prior experience and references (if available).
That being said interviewing and hiring is not everything. It takes a good 1-2 years before someone has deep context of the codebase to starting being really effective and having a long term view.
As for actual performance, mentorship makes a huge difference. It’s crazy how little companies invest in this. I’ve learnt a ton by working with folks who have more experience than me and them telling me why a certain approach is the wrong way to think about something, and what is the right way to go about it. What skills I should be focusing on, how I can make a bigger impact. The amount of money and effort spent on hiring vs mentoring blows my mind.
Ask yourself this question: How many minority folks in your org have been promoted to senior levels vs hired?
Great article! I remember finance/hedge fund recruiting from college. Looking back, it is absurd for buy-side companies (whose entire business is to develop conviction of unconstrained markets/businesses) to ask hyper-constrained coin-flipping probability riddles as a filter.
One time when I got a coin-flipping question, I told the recruiter that there was precisely zero signal in the question, and I then offered up a few alternates that he could use for other candidates. (Happily, I did not get the offer!)
From talking to various folk about this, it has become clear to me that multiple interview setups is probably ideal. Personally I find the four 1-hour bits significantly luck-based, even if designed well. Lots of noise. Though I did read a book a long ago that I remember had a pretty good plan for reducing that noise (Hire with Your Head). I am finding myself more fond of the project-based setups, though I know some people personally who absolutely can not do a "homework assignment", which is fair enough. There's a venn diagram at play between what the interview is measuring and how good the candidate is, and different interview setups will intersect with "quality of candidate" in different ways.