For years I have been warning that “scaling” — eeking out improvements in AI by adding more data and more compute, without making fundamental architectural changes — would not continue forever. In my most notorious article, in March of 2022, I argued that “deep learning is hitting a wall”. Central to the argument was that pure scaling would not solve hallucinations or abstraction; I concluded that “there are serious holes in the scaling argument.”
And I got endless grief for it. Sam Altman implied (without saying my name, but riffing on the images in my then-recent article) I was a “mediocre deep learning skeptic”; Greg Brockman openly mocked the title. Yann LeCun wrote that deep learning wasn’t hitting a wall, and so on. Elon Musk himself made fun of me and the title earlier this year.
The thing is, in the long term, science isn’t majority rule. In the end, the truth generally outs. Alchemy had a good run, but it got replaced by chemistry. The truth is that scaling is running out, and that truth is, at last coming out.
A few days ago, the well-known venture capitalist Marc Andreesen started to spill the beans, saying on a podcast “we’re increasing [graphics processing units] at the same rate, we’re not getting the intelligent improvements at all out of it” – which is basically VC-ese for “deep learning is hitting a wall.”
Just a few moments ago, Amir Efrati, editor of the industry trade journal The Information further confirmed that we have reached a period of diminishing returns, writing on X that “OpenAI’s [upcoming] Orion model shows how GPT improvements are slowing down”.
Just as I argued here in April 2024, LLMs have reached a point of diminishing returns.
§
The economics are likely to be grim. Sky high valuation of companies like OpenAI and Microsoft are largely based on the notion that LLMs will, with continued scaling, become artificial general intelligence. As I have always warned, that’s just a fantasy. There is no principled solution to hallucinations in systems that traffic only in the statistics of language without explicit representation of facts and explicit tools to reason over those facts.
LLMs will not disappear, even if improvements diminish, but the economics will likely never make sense: additional training is expensive, the more scaling, the more costly. And, as I have been warning, everyone is landing in more or less the same place, which leaves nobody with a moat. LLMs such as they are, will become a commodity; price wars will keep revenue low. Given the cost of chips, profits will be elusive. When everyone realizes this, the financial bubble may burst quickly; even NVidia might take a hit, when people realize the extent to which its valuation was based on a false premise.
§
The sociology here has been perverse too, for a really long time. Many people (especially LeCun, but also a legion of tech influencers who followed his lead) have tried to deplatform me.
The media has done little to counter the mob psychology; they have mostly listened to people with money, with vested interests, not to scientists. Many of us, including Melanie Mitchell, Subbarao Kambahapati, Emily Bender, Ernie Davis, etc. have been emphasizing for ages that there are principled limits with LLMs. Media (with notable exceptions like Ezra Klein, who gave me a clear platform for skepticism in January 2023) has rarely listened, instead often glorifying the hype of people like Altman and Musk.
Worse, the US AI policy now, and likely in the next administration, has largely been driven by hype, and the assumption that returns for LLM scaling would not diminish. And yet here we are at the end of 2024, and even Altman and Andreesen are perhaps starting to see it.
Meanwhile, precious little investment has been made in other approaches. If LLMs won’t get the US to trustworthy AI, and our adversaries invest in alternative approaches, we could easily be outfoxed. The US has been putting all its AI eggs in the LLM basket, and that may well prove to be an epic, massive mistake.
§
In April, when I first saw direct empirical evidence that this moment had come, I wrote (and stand by):
If enthusiasm for GenAI dwindles and market valuations plummet, AI won’t disappear, and LLMs won’t disappear; they will still have their place as tools for statistical approximation.
But that place may be smaller; it is entirely possible that LLMs on their own will never live up to last year’s wild expectations.
Reliable, trustworthy AI is surely achievable, but we may need to go back to the drawing board to get there.
I’m glad that the market is finally recognizing that what I’ve been saying is true. Hopefully now we can make real progress.
Gary Marcus has been warning about the foundational limits to traditional neural network approaches since his 2001 book The Algebraic Mind (where he first described hallucinations), and amplified those warnings in Rebooting AI and his most recent book Taming Silicon Valley.
