Monday, October 06, 2014

Gaming the Coding Interview

Paul Graham's essay on how you can't really game startups had me thinking about the coding interview. Google had a lot of studies showing that the interview as practiced by Google wasn't very effective: in other words, interview scores don't really correlate with actual job performance. In part, this is because Google's not a startup any more --- political ability probably determines your promotions and effectiveness within Google than simply being good at engineering. But a major part is also that the coding interview is very susceptible to being gamed.

For instance, if you read Cracking the Coding Interview and were diligent about it (i.e., actually worked through the problems and practiced at them), you'd stand a good chance of doing really well during Google's interview process. Lest you think that this is a recent phenomena, even in 2003, Google's interview process was very similar. I remember being asked to reverse all the words in a sentence, and a few other puzzler type questions, and even during my interview, I remembered one interviewer telling the next one as the hand-off was happening, "this guy knows all the standard interview questions." Back then, Gayle's book didn't exist, but 10 years of interviewing for startups and interviewing at startups had hit me with every interview question that could be easily covered in a 45 minute session.

I will note that Facebook does have tougher interviews today than Google (they're hiring slower and therefore can be more picky), but from what I've seen their interviews are no less subject to being gamed.

When I look back at the interviewing process, there's really only one company that's stood out for having an interview process that couldn't be easily gamed, and that's Wealthfront in late 2012. I only include the date because in between, startups can change a lot and for all I know they could be interviewing like Google today.

The way Wealthfront conducted their interview was by pair programming. The candidate would come in, and pair program real problems with their "interviewer". The experience is intense, and in many ways eliminates the possibility of hiring someone who couldn't even write correct java syntax, or construct unit tests for code he'd just written. It's a good way to go and difficult to game, since you have to actually be able to design, structure, and turn ideas into code all the way to the testing and debugging steps.

Another good idea I've seen at certain startups is to put the culture fit interview first, before any technical interviews get done. The reason for this is if you get a candidate who's stellar on the technical side, it's actually very difficult to reject him for cultural reasons. I can attest to this, as one of my early hires at Google bombed out precisely for that reason, though without doing much damage. By putting the cultural fit interview first, you eliminate the bias to hire, even though you might waste a bit of time.

Post a Comment