How well do LeetCode ratings predict interview performance? Here's the data.
This blog post was originally published on interviewing.io and has been republished here with permission.
Have you ever wondered if you should spend more time on LeetCode, participate in those contests, or focus on solving harder problems? A popular Reddit post suggests you need 700+ questions and a LeetCode rating between 1800-2000 to pass FAANG interviews. Is this really what the data supports? To answer these questions and more, we looked at our users' LeetCode ranks and ratings and tied them back to interview performance on our platform and whether those users worked at FAANG.
In this post, we’ll share what we’ve learned.
The experiment
interviewing.io is an interview practice platform and recruiting marketplace for engineers. Engineers use us for mock interviews. Companies use us to hire top performers. Hundreds of thousands of engineers have used our platform to prepare for interviews, and we have performance data for over 100k technical interviews (split between real interviews and mocks).
We surveyed almost 700 of our users and asked them to share their LeetCode and LinkedIn profiles. From those profiles, we pulled our users’ employment history, as well as their LeetCode data: number of problems worked, ratings, and, if they had it, contest performance. Finally, we cross-referenced all this data with their performance in mock and real interviews on interviewing.io.
This data set allowed us to start asking interesting questions. How much LeetCoding is useful? What kinds of problems are most useful to practice? How does LeetCode performance relate to performance in interviews with real people? Do LeetCode contest scores predict interview performance, and is a competitive coder likelier to work at a big tech company? Are hard questions worth solving or should we just stick with mediums? Let's find out!
Results
The matrix below summarizes our findings. In addition to all LeetCode fields listed below, we also looked into global ratings and contest ratings. It was harder to find a large enough sample size to see any effect of those who did contests. Since the results were not large enough to be informative we have excluded them from the rest of this piece. All findings listed below are statistically significant, with lighter squares indicating stronger correlations.
For each profile, we looked at the following attributes:
Total questions: the total number of questions solved on LeetCode
Hard questions: the number of "hard" questions solved on LeetCode
Medium questions: the number of "medium" questions solved on LeetCode
Easy questions: the number of "easy" questions solved on LeetCode
Worked at FAANG: whether or not the user has ever worked at a FAANG company
interviewing.io percentile: how an interviewing.io user stacks up against other users of the platform, after having completed at least one interview
How LeetCode attributes relate to having worked at FAANG and performance in mock and real interviews on interviewing.io. Correlations range from 1 (directly positively correlated) to -1 (directly negatively correlated). 0 means there’s no relationship.
This matrix summarizes the correlations between LeetCode attributes (y axis) with 1) whether people worked at a FAANG and 2) how well they performed in interviews on interviewing.io (x axis). The higher the number (and the darker the color), the stronger the relationship.
For instance, the number of questions a user solved correlates with working at a FAANG company and a user’s percentile ranking on interviewing.io. We see that it is a stronger predictor of interview performance than it is for working at a FAANG company (0.27 and 0.17, respectively).
Let's dig into the most exciting findings!
The total number of questions you have completed matters!
Unsurprisingly, the total number of questions a user has completed correlates with having FAANG on their resume and doing well in technical interviews. Those who work at FAANG companies appear to have completed more questions than those who do not.
FAANG engineers do more total questions than non-FAANG engineers
Don't fret, though, as this doesn't mean you're doomed to be behind people who have started sooner and have amassed thousands of questions. The vast majority of our top users stop LeetCoding after they've hit about 500 questions. Only a fraction of top users have done more than ~500 questions, and only about 10 did more than a thousand questions. As you might expect, there are seriously diminishing returns associated with doing more than 500 questions. Those who continued past the 500 questions performed only marginally better than those who had stopped near 500.
“In the zero-sum interviewing game, those who complete the most LeetCode questions generally get the highest interview scores and work at the most desirable tech companies, but completing ~500 questions will put you among the top talent in the current market.”
The difficulty level you attempt matters, too!
It is common wisdom on the LeetCode discussion board and subreddits to "do medium questions, not hard questions," with the argument that companies don't ask LeetCode hards. Does this advice stand up to scrutiny? Mostly no.
The data shows a clear bias towards those who solved more challenging problems, the exact point at which this matters is up for debate. At interviewing.io we find that before tackling difficult questions, you just need to first start doing problems. Getting started matters more than attempting to complete a particular question difficulty, so don't focus arbitrarily on medium/hard questions, just focus on practicing!
Tip 1
Those who solved more challenging questions needed to do far fewer questions to have the same chance of getting into a FAANG company
Those who solved more challenging questions needed to do far fewer questions to have a high interviewing.io percentile
Before tackling mediums and hards, we need to first get good at easy questions. Once we can tackle mediums, for each additional 50 questions you complete, you increase your score of passing your interviews by three percentage points.
On the other hand, a LeetCode hard carries over twice the benefits of a medium question. Completing 50 hard questions increases your interview score by seven percentage points! Our data shows that you'd need to complete close to 233 medium questions to get the same benefits that 100 hard questions would give. Consider these two people (fictitious but inspired by real examples in our data).
Notice how Kara had to do far fewer questions because they focused on hard questions rather than medium questions
These two people are statistically likely to have similar interview scores, yet Kara completed half the number of questions Alex did. While it goes against standard advice, harder questions having a higher benefit make sense. A LeetCode medium tends to require you to do one thing (perform a DFS, scan an array, etc.) to get the correct answer. In contrast, hard questions require you to do multiple things (memoize results while performing a DFS, tally prefix sums while scanning an array, etc.). The compounding nature of multiple tasks in harder questions necessitates deeper learning in fewer questions.
Admittedly, there is a chicken-and-egg scenario going on here. Naively, one might think everyone should just do hard questions, but that doesn't work in practice. Jumping straight to hard questions is an infeasible leap for most people, and the reason many people can complete hard questions is precisely because they have finished many medium questions first. Our advice is not to jump to hard questions immediately. Instead, you should be mindful and not stay at the medium difficulty level longer than necessary. It isn't that the advice to "do medium questions" is wrong—it is just overused. It is natural to want to be able to complete a question regularly in a short period of time. Still, if you find yourself completing LeetCode mediums quickly, it indicates that you have stagnated and would be better off moving on to hards. If you have solved 100 or more medium questions, you'll likely get more bang for your buck by transitioning to hard.
Tip 2
Finally, we must also acknowledge that not all hard questions are created equal. This tip is my gut feeling, and we don't have data to back it up, but intuitively, we've all done hard questions that seem impossible, hard questions that seemed like they should have been rated medium, and even medium questions that seem like they should have been hard. Question difficulty is somewhat subjective, so if you focus on hard questions, choose questions with a high acceptance rating and/or ones that are "frequently asked." These questions tend to be achievable (they don't have a single impossible trick you need to just have known), realistic (they don't test nonsense algorithms no one knows like Manacher’s algorithm), and solution-flexible (there is often more than one valid approach).
Tip 3
Contests and rankings don't matter
Surprisingly, we found no correlation between LeetCode ratings and their interviewing percentile. Those who were great at contests also didn't appear more likely to have FAANG on their resume. Two confounding variables that might have caused these unexpected results are:
Small contest selection size: Despite our best efforts, we received a relatively small group of candidates that attended any number of contests, let alone several. The data set may be too small to find a correlation.
Biased candidate selection: We incentivized candidates with free interviews, so those who regularly complete contests (and therefore have a lot of practice and likely feel confident in their skills) might not be interested in the rewards and weren't studied in this experiment.
Tip 4
Conclusion
In the end, LeetCode questions are a reasonable proxy for predictors of interview performance. While ratings and contest scores don't seem to matter[1] questions —and question difficulty—do. Focus on tackling increasingly difficult questions until you've hit that point of diminishing returns near ~500. Don't stagnate with medium questions; pick popular and frequently asked hard questions, too. And don't stress if your friend has a higher LeetCode score than you do; turns out it doesn't mean much!
Footnotes:
Footnotes
A caveat here is that, strictly speaking, we're reporting correlations—not necessarily causal relationships. This leaves open the possibility that what we were measuring was not exactly a return to LeetCode but rather the effect of some third variable that increases both LeetCode activity and job prospects. For example, say MIT has a course that emphasizes coding puzzles, and MIT grads also get better jobs. In that case, what might look like a LeetCode benefit is actually an MIT benefit, and controlling for that would make the apparent benefit disappear. At the very least, though, these patterns are informative about the typical profile of successful engineers, which is likely a useful thing to emulate in general. Also, remember that our data still shows solving a healthy number of questions regardless of their difficulty can boost your chances of getting through difficult interviews. By definition, many people can pass interviews with far fewer problems completed, and many will pass after doing more problems. ↩