The Return of a Bad Engineering Metric: When AI Token Usage Becomes Performance Theater

When CEO of Nvidia, Jensen Huang, says, “If your $500K engineer isn’t burning at least $250K in tokens, something is wrong,” it brought a lot of people’s attention to the new paradigm that token consumption has become a key metric in AI adoption. Although it’s from the guy selling shovels in a gold rush saying you need to dig more, there’s some truth in how companies are evaluating their own developers.

Some companies evaluate developers based on whether they have used up their token quota, and some companies focus on whether there is a “lack of” token usage for certain developers, which may be a signal that this developer is not keeping up with the trend of AI development. The token quota for developers has even become a benefit, or status symbol for some developers, signaling that they are doing more work than others and being more AI-adept than their peers.

All of this is reminiscent of how some companies in the 1970s and 1980s used a line of code to evaluate developer performance. It has the same simple, direct, and easy-to-understand relationship to how much work is being done, and it’s accurately quantifiable for everyone. But as Bill Gates once famously said: “Measuring programming progress by lines of code is like measuring aircraft building progress by weight,” and once a measure becomes a target, it basically ceases to be a good measure. Developers can easily write bloated code to meet their quotas.

Evaluating work or the level of AI adoption by token usage has almost the same problematic pattern. It’s extremely easy to bloat token usage by paralleling agents to do simple tasks and just call it being careful and thorough. Modern, larger LLM models have a high tendency to over-engineer, writing lines of code that look professional but serve very little purpose in terms of the actual context of how the code is being used. KPI like this will certainly encourage developers to both intentionally and unintentionally write code that is costly to maintain.

Like the cost of maintaining an application with a high line of code, high token consumption is a very direct cost to the companies. If team members start to compete on how many tokens they consume, it will most certainly drive up development costs. Or even make some relatively simple work a lot more complicated than it should be. Worst of all, AI-assisted development tends to accelerate the growth of one’s code base size. Whether it’s a result of higher productivity, or a result of AI over-engineering, or even developers intentionally bloating their output, a larger code base will always cost more to maintain, and as multiple studies have shown, AI coding quality deteriorates as the relevant code base gets larger.

AI tokens have all the characteristics of bad metrics, like lines of code. Consuming more tokens is not a direct indication of the developer using AI in a more sophisticated way, and the lack of token usage does not necessarily mean that the developer is not keeping up with the age of AI. Actually, in most cases, the nature of the problem the developer is trying to solve determines how many tokens should normally be consumed.

A more practical approach that some managers are taking is to simply not take AI token usage into account. If a developer actually benefits from AI, the performance numbers already in place, before AI, should be a clear enough indication that AI is helping. All AI benefits should be measured as actual performance results from a developer, whether the AI is helping or not.

A developer using less AI to achieve the same result should not be punished just because they used less AI. Right now, many managers do not reward doing the same amount of work with fewer tokens, but as AI-assisted development becomes more common, adopting AI does not require an extra incentive. In the long term, it will make sense to encourage developers to be smart about their token consumption and be AI cost-conscious with their work.

Currently, the trend of burning as many tokens as one can is being fueled by the massive subsidies from AI model vendors, making AI subscriptions significantly cheaper than their API call counterparts. Most experts believe this is not a sustainable model, and the AI token cost will eventually need to be reflected more clearly when the subscription subsidy gradually goes away.

When the cost of a token becomes closer to reality, development teams will start to realize how unreasonable it is to use token usage as a performance metric. The performance metrics that were already in place for developers should be the actual bar to measure developer performance. In the end, using AI is always just a means to the end, not the end itself.