As organizations everywhere work to accelerate their digital transformations, they are seeking to find software engineers to power these large-scale tech initiatives. Karat's 2023 Hiring Trends Report reveals the demand for this kind of top tech talent makes these professionals all the more valuable. In fact, the report features a Harris Poll survey of nearly 800 engineering and talent acquisition leaders, and more than half of them believe that strong engineers are worth at least three times their total compensation.

That sentiment has increased by over 20% since 2020, and the team at Karat sees that trajectory continuing. Recent developments in artificial intelligence (AI) have given us a tantalizing glimpse of how software can be used in domains that seemed like science fiction a few short years ago. The digital transformation across many sectors of industry is just beginning, and it will continue to fuel the demand for software engineering talent for years to come.

Rather than replacing software engineers, we're seeing how the adoption and use of these AI-powered tools serves to make fundamental software engineering skills even more valuable. Here are some key insights that we've gained after leading more than 300,000 technical interviews for customers around the world.

Why Foundational Skills Are Fundamentally Important

As the world's largest technical interviewing company, Karat partners with Fortune 100 and 500 companies to identify highly qualified engineering talent for a variety of roles, including those with an AI focus. Through this work, we've discovered that regardless of role, our customers want to hire candidates who have strong, fundamental software engineering skills. Additional expertise in specific engineering disciplines, including AI and machine learning, is certainly valuable; but they see those skills as supplemental.

The reason: most of our clients use the same proficiency bar to evaluate the skills of every software engineer in their organization. AI and machine learning engineering expertise are measured separately as additional skill sets. Furthermore, our customers also gauge those additional skills without using AI.

Findings in Karat's Hiring Trends Report support this approach. Insights from leaders at Tesla, Walmart, Okta, and Airbnb show that the most successful engineering and talent acquisition leaders are seeking out candidates who demonstrate exceptionally strong problem-solving skills. Proficiency in specific languages and technologies matters less than the capability to successfully resolve unique and unexpected obstacles. In most cases, these leaders expect software engineers to be able to resolve open-ended, complex challenges that may have more than one solution. This problem-solving focus helps hiring leaders understand how engineering candidates would overcome those obstacles in the face of the real, ambiguous, and unpredictable circumstances they'll find on the job.

How to Identify Foundational Skills When Hiring Engineers

One reason some people look toward more specific skill sets is that foundational skills can appear difficult to identify and evaluate during the hiring process. It may even seem like some of them aren't quantifiable in an interview. There's a grain of truth to this idea: Hiring processes that are front-loaded with resume screens and candidate interviews have a harder time getting a clear signal of a candidate's foundational skills. Instead of assessing the candidate's capabilities, these traditional processes perpetuate pedigree bias because they rely on past work experience and education credentials - neither of which create a strong signal to hire or measure a candidate's ability to do a job effectively.

At Karat, we recommend refining your hiring process to focus on skill proficiency in the initial phases (Figure 1). This can be accomplished without adding additional rounds of interviews, which can distract your very busy engineering teams and compromise the candidate's experience. Using adaptive technical assessments to verify candidates' skills early in the process can help your organization:

  • Focus on whether a candidate has developed the skills necessary to successfully complete day-to-day work associated with the job.
  • Reduce the amount of time talent acquisition and engineering teams spend reviewing resumes and interviewing candidates, resulting in a simpler and more seamless way to hire.
  • Convey a sense of respect and value for the high-quality talent that you're competing with other organizations to hire.
Figure 1: Adaptive technical assessments make hiring efficient
Figure 1: Adaptive technical assessments make hiring efficient

Most importantly, our research reveals that companies who use technical assessments to advance large candidate pools through several initial phases of the interview process result in higher-quality hires. Adaptive technical assessments, likely administered online by a third-party provider to eliminate reliance on take-home tests and other coding exercises, relieve the burden on teams that may already be overtaxed or over-extended. It also helps you “right-size” your candidate pools - creating a deep reservoir of qualified talent for interviews.

Traditional interviews have their place, but that place is later in the process to help narrow the pool of qualified candidates down to the best ones for the role.

How to Detect Use of AI to Compensate for Skill Deficiencies During Interviews

In today's highly competitive talent marketplace, candidates are looking for any way to show their best selves in the interview process. With AI tooling, some candidates may cross the line, from putting their best foot forward to giving the appearance of skills they don't actually possess. Here are a few key indicators that a candidate you're interviewing may be using AI to inflate their skill level during an interview:

The candidate struggles to explain the rationale for their solution to a challenge. If the candidate is unable to explain in detail what the code they're writing does and why they've written it, it's likely that they're getting a little extra “help” from an outside source, such as ChatGPT. While qualified candidates can and do make mistakes in interviews, it's far more common for them to know what they are trying to do but make some errors along the way while working quickly. Explaining code rather than simply writing it also allows candidates to showcase the problem-solving skills that engineering and recruiting leaders are looking for in top candidates.

Figure 2: AI models often fail because they cannot understand the code. In the case above, ChatGPT is trying to convert types to resolve an error when the real problem is that it is operating on the wrong variable. In the same way, candidates who are overly reliant on these systems often fail to adapt their code. This results in needing to rewrite it entirely to account for small changes.
Figure 2: AI models often fail because they cannot understand the code. In the case above, ChatGPT is trying to convert types to resolve an error when the real problem is that it is operating on the wrong variable. In the same way, candidates who are overly reliant on these systems often fail to adapt their code. This results in needing to rewrite it entirely to account for small changes.

The candidate struggles to adapt their solution to account for small variations in the problem. Experienced software developers know that sometimes requirements change. Many good interview problems have small variations, such as looking for all valid answers instead of just one or imposing an additional restriction on the answer. Asking a candidate to discuss how their code would change to account for a variation can help them concretely demonstrate their understanding of how the components of their solution function. Candidates who have enlisted the help of generative AI to arrive at their solutions often find themselves unable to deal with these on-the-fly changes or propose to start from scratch and build a totally new solution.

The candidate's explanation of their code doesn't match what the written code does. Writing software is more than just producing code. It's also communicating with teammates throughout the process and being able to share the underlying concepts behind the code. This communication is why it's essential to involve existing software developers in the interview process: the best way to evaluate whether a candidate can explain their code is to ask them to explain their code to a qualified - that is to say, technical - audience. Candidates who are overly reliant on generative tools often struggle to explain how each of the pieces of the code contribute to the whole. This is contrary to how most software developers work, with a deep understanding of the individual pieces. A software engineer who wrote their own code is more likely to get lost in the details than to be unable to explain them!

It's not that real developers, even good ones, don't fail. It's that even their failures show us how they're thinking. These indicators aren't direct tests for AI-enabled cheating - they're core assessments of job-related skills that AI still can't help with. The advent of AI-enabled cheating has just made it more important to focus your interviews on the core problem-solving work of software development, and not use nearby proxies like the ability to generate syntactically valid code.

From our experience interviewing thousands of candidates, before this transition and through it, we see that the adoption and expansion of AI-powered tools into software engineering is not as destabilizing as it may initially appear. These tools are likely to change the tech industry and the way we work, but we've lived through changes of this magnitude. Google ushered in such a change with the introduction of the first truly effective search engine. Stack Overflow broke down internal silos around technical knowledge. Like AI, both these tools changed the day-to-day work of nearly everyone in software, but they didn't diminish the need for the fundamental skills of software engineering. By focusing your tech hiring on these foundational skills, you can build a team that's resilient not only to today's disruptive technology, but to the next disruptions coming down the line.