I think the author is mostly right about the current state of AI, but his future predictions (or worries) are based on a false premise: that the massive LLMs will keep improving in the future.
As far as I have seen the improvements have clearly slowed down, while the energy consumption is rising linearly (or worse). It’s like the energy (money) vs. performance graph is logarithmic, and the companies are expending double the energy to get a 10% improvement. Something like that is not sustainable, and the money seems to indicate so.
I really think that LLMs are a dead-end for AI. A really useful dead-end, once the bubble pops and with time, we get a useful working model for them, probably based mostly on local LLMs, maybe using specialized training data.
Energy efficiency has improved by orders of magnitude - leading to much higher energy use. It’s the Jevons paradox and it’s as old as coal-gas lighting. Last year some guy recreated GPT2 for twenty bucks. Corpus to model in one hour. OpenAI never said how much the original cost, but there was at least one comma.
But yeah, LLMs are fundamentally limited, because ‘what’s the next word’ shouldn’t work. The fact it’s accidentally this flexible and powerful, even with its many infamous fuckups, is a reminder that neural networks in general will permanently alter computing. Models trained on supercomputers can run on any potato. Any problem with good examples can be addressed, without first being solved.
LLMs are fundamentally limited, because ‘what’s the next word’ shouldn’t work.
Yes, you’re right. However, for fear of coming off as an AI sycophant (I’ve yet to sacrifice my brain at the altar of our future AI overlords), LLMs aren’t the whole picture. Plenty of research is dedicated to essentially combining the best of each class of AI algorithms into a composite model of intelligence. For instance, “Neuro-Symbolic AI” is really just the result of giving an LLM (good at translation, search, synthesis, bad at symbolic reasoning) a symbolic inference engine like Prolog (good at symbolic reasoning, no native ability for translation/search/synthesis). I’ve been coding for over 20 years, and I’m impressed at its results for software development.
This all is reminiscent of Moore’s Law; even though we keep running into the physical limits of CPU clock speeds, transistor size, etc. we keep finding clever ways to work around those limits.
Of course I’m not saying we should; these models are, after all, models of intelligence, not wisdom.
Edit: fix apostrophe splice
I’d be curious why you think LLMs are dead ending? Is it that you think the Jepa models are likely to find success and win out or do you think the LLMs in generally are just hitting their peaks?
Your point on power usage is interesting, although I think that is mostly on training not usage correct?
I think they’re a dead-end mostly because of the exponential cost vs. performance. The decreasing returns are obvious, and the companies are trying to adapt by raising token prices, but that will not be enough with the current user numbers (or even double or triple, if we believe the analysts). I think that, at least with these large LLM companies, we’re actually beyond the point of economic equilibrium with this technology, at current energy and water prices.
And yes, training is more expensive than usage. That’s probably the reason why Anthropic suggested a pause in LLM development (training), supposedly because of the fear that AI could become Skynet, but really because they are getting an IPO soon and if people see their current balance numbers, the IPO would fail and the bubble would probably pop. Which really proves my point a little: the economics of these companies “improving their LLMs” (training) don’t make sense at current energy prices.
Well frankly i think the field of eco terrorism is about to expand greatly so maybe start preparing for that.
I’m still employed and I see myself employed (at least in that company) for a foreseeable future.
Hmm, well, ok. Moving on then
Try to argue with the LLM evangelists, the inevitable brain damage it causes will let you get on disability.
On one hand this is not the first hype cycle, on the other hand the other hype cycles didn’t all fade. The inevitable bullshit brought on by vibecoding and shit like that is eventually going to be some kind of problem. People may or may not just ignore that problem, like most other issues in tech.
Give it time. My software career is also affected. At the rate they’re spending money at an order of magnitude higher than they’re making. They’ve also all borrowed money from each other. It’s going to collapse in a big heap. Hopefully before it sucks in mum and dad investors.
It’s gonna be about a decade more.
I’ll be retired by then.
Not sure I agree with the “three pillar” framing in this one. I think it’s more like he was able to use the LLMs effectively because he had already built those years of experience. Someone new trying to vibe code their way into designing robust distributed systems is going to need a similar amount of time to build the correct intuitions, because the LLM doesn’t give a shit one way or the other, it’ll just do what’s in the prompt.
On the other hand, commodotizing domains that fit a repeatable pattern isn’t so bad, is it? At work I sometimes have to remind people that we’re generally just building CRUD apps to interact with data, which isn’t exactly at the bleeding edge of software. I like the idea of software engineers getting to be more portable, but unfortunately the industry still thinks LLMs are going to replace most jobs, which is obviously not possible.
The world is changing. It happens from time to time. In this case the change is a particularly big one and it’s still ongoing, so I can’t make any predictions about where it’s going to end. But I can be pretty confident that it’s not going to magically change back. So my best advice is to try out the new tools, see whether you can adapt to them and use them to improve your own productivity in new ways, and if not then as a fallback start looking at other directions to take your career.
Harsh, perhaps, but the world does as the world does.
It’s a tool, for me atm it’s like spellcheck and google had a baby
We were taught that generalists and specialists will always have their roles.
Well, and therein lies your actual mistake. Believing what you were told by higher-up about the irreplaceability of the human intellect. Economists have long made such claims sothat they could motivate enough people to go through the strenuous tasks of learning and developing knowledge, meanwhile it was always clear that the development of automatic intelligence was only a question of when, not if.
