~2 min read Goodhart's Law ("When a measure becomes a target, it ceases to be a good measure") is something that more people in software engineering leadership really should know. "Our engineering team spent $1 million on tokens last month." You may have heard someone say that. The number may be higher or lower. It might be $10k or $100k. The point being is that usually it's some number that's large enough in that business to suddenly be among the largest single line item. There's both anecdote and metrics to argue for it that it did happen. Like the below graph from Ramp, where you can see the clear inflection point where Claude Code took over. That should be astonishing to you, if you're a sane person. Most companies don't have a large bucket to spend on any one kind of service or product. So why did it happen? It happened because for a short period of time, about Jan to June 2026, there existed the idea that token maxing, maximizing the number of token used per day, was a measure of productivity. We had the CEO of Nvidia talking about it, "If that $500,000 engineer did not consume at least $250,000 worth of tokens, I am going to be deeply alarmed," But unless, you're like me, an engineering manager, you've not likely been asked to prove that this tool is worth it. This ROI productivity ask is not new to AI. Executives have regularly asked for this from me and regularly reached for silly things like tickets closed or lines of code. However, you may have heard number of PRs be tossed about as the new one. It's probably a good metric to show the inflection point of pre and post agentic coding work. Most places I talk to see somewhere on par of 3x-10x more PRs per week. All that's telling you is that now the bottleneck has moved downstream from code creation to code review. And so we come back to it here: software engineering has no productivity metric. Not because software engineering is special, but because creativity itself isn't really quantifiable. Every function has this problem in a software company. Design. Product. Marketing. Sales. And yes, even the money people in accounting and finance. But I'd go further an argue that most companies have this "problem" of not quite knowing where the magic is coming from. Software engineering is the only place where it's generally asked, because we have the highest amount of spend on people with the least direct tie to how it generates money. Everyone knows it does, because without these people the software work doesn't get done and there's nothing to sell people. The moment we figure out software engineering "productivity" is when we also figure out something else about our nature of our existence. I tend to not want to foreclose on things like this being possible. But I hope we don't. — Robert This message is 100% organic human effort, so it will have typos. No, I didn't send it through an LLM to typo check it. It was made with just my fingers and my eyes. |