"Movement is the main thing" is precisely why pursuing compute-in-RAM makes some sort of sense to begin with. But DRAM fabrication processes are quite specialized and do not perform well with pure compute logic. The overall profile of this thing will arguably be similar to a rather weak NPU, though with much better memory bandwidth - one key limitation, as with NPUs, will be the bespoke programming model and lack of support for the latest compressed/quantized number formats, which heavily limits the usefulness of being able to access memory directly. GPUs, even weak iGPUs, can dequantize/pad parameters on the fly which adds a lot of flexibility - and expose standard, well understood compute capabilities via CUDA, Metal or Vulkan. This is not quite comparable unfortunately.
If what we want to do with this is make cheap QKV sweeps, then "a weak NPU with a lot of mem bandwidth" seems good enough? Exactly the tool for that job, and nothing else.
Also spares us the trouble of dealing with weights. By the time we're in QKV realm, the weights have already weighted.
> Small models are rapidly growing in capability, require less compute to train and serve
According to Jevons' paradox a reduction in resource requirements (improved resource efficiency for the same payoff) leads to an increase in demand. This stops working when demand for compute is completely exhausted, but we are very far from that. There's even some very silly predictions floating around (see the latest Dwarkesh Patel podcast) that say compute will soon be most of the economy, even dictating market interest rates. Now, that has to be wrong, but the directional outlook is closer to correct than "very small and efficient models mean there will be ~0 demand for HPC-like compute".
I don’t think this line of thinking is particularly robust because it ignores how AI is being used and where the resource usage is coming from.
Right now there are a small number of very very resource intensive use cases that are being subsidized by OpenAI and Anthropic. There are people generating millions of lines of code because it’s basically free at the point of use, despite the code producing very little value. Anthropic and OpenAI frequently “reset” customer limits to allow them to use even more resources at no additional cost.
The majority of use cases across business are not generating millions of lines of code per employee. The majority of businesses need just a little bit of automation to radically improve the way they operate. A software engineer making endless projects because it’s free to do so might use hundreds of billions of tokens per year, but an entire manufacturing business could be revolutionized with a few million tokens per year.
I think 2 things can be true:
1. There is very little penetration of AI across the economy and huge room to grow in the number of businesses deriving economic value from AI
2. The compute usage today is vastly overrepresented by usage outliers who are not paying the cost of their usage and will stop when forced to pay the cost
We could see AI usage 10x while seeing compute decrease 10x if the type of usage shifts. Most businesses just need smarter macros.
> Anthropic and OpenAI frequently “reset” customer limits to allow them to use even more resources at no additional cost.
Surely this applies to fixed-price subscriptions, not per-token spend? Large enterprises (the "very very resource intensive" large-scale users) have to pay per token.
A lot of companies avoid paying for usage by encouraging their employees to use individual subscriptions. Outside of the short lived tokenmaxxing fever dream, enterprises are conscious of their usage with companies like Uber and Amazon reigning in their usage massively and companies like Ramp building their own routers for cost minimization.
Facebook is reportedly the company that spent $500 million in a single month on tokens. There are individual non-enterprise users rotating multiple subscriptions incurring $10k+ in tokens per subscription. Facebook’s $500 million month… is equivalent to ~10k individual subscriptions which could be as little as a few thousand of the heaviest users. That’s $500 million when billed on usage, or ~$2 million on plans.
The reason resets are such a big deal (people have set up websites to track them, tweets announcing them get millions of impressions) is because there are huge numbers of users pushing their plan limits every single day. If there was huge demand from usage-based customers (the large enterprises) that OpenAI and Anthropic couldn’t meet, they wouldn’t be handing out resets like candy.
I think a realistic belief is that Anthropic and OpenAI have vastly overstated demand and are using resets as a way to keep usage artificially inflated at a substantial financial cost. I’d guess fixed price plan users make up at least 95% of usage.
Jevons paradox is a veridical paradox, which, as you said, means that it's a true statement that merely looks wrong because it is counterintuitive.
I know that some people think that the word "paradox" should be only used to refer to antinomy paradoxes which are often called "true paradoxes" (such as "this sentence is false") which lead to a contradiction without requiring a flaw in reasoning.
Yeah, I think there's a tendency to underestimate how much demand is still gated behind cost constraints. The market for this is HUGE.
The PC era, call it 1975-2005, was one of the greatest wealth creation events in history, was characterized by the cost of the underlying commodity dropping mercilessly for the whole time. Each time it did, the space of problem you could solve with a PC would increase, to the point that by the end, they were both replacing mainframes and powering users who do nothing but chat and post cat pictures.
Could there be a correction in the short run? Quite possibly. I think an underestimated last mile problem is just the massive weight of bureaucracy and human process inertia. But in the long run, cheap, efficient intelligence is a new engineering capability that we've just begun to even explore.
> how much demand is still gated behind cost constraints. The market for this is HUGE.
I think this misses the actual limits here.
The problem isn't demand it's, "how much people are willing to spend on it".
Cheap AI has to be served on cheap compute, and if inference gets cheap enough to unlock massive usage numbers, by definition it also doesn't require anywhere near as much infrastructure per unit of demand.
Take DeepSeek serving ~100T tokens/day, depending on workload and utilization, you're potentially talking about only a few thousand last-gen GPUs. With current-gen GPUs maybe closer to ~1,000, and with Rubin even fewer I will be damned if I could get my hands on one.
That's the part I think people are missing when they extrapolate token demand into enormous infrastructure or AI revenue.
Yes usage will explode. But if the cost per unit collapses, the revenue doesn't necessarily go up with it.
You can't simultaneously argue that intelligence becomes so cheap that everyone uses enormous amounts of it, while also assuming customers will somehow spend trillions of dollars a year consuming it.
There is no obvious $1T customer-facing AI revenue number at the end of this rainbow in the short/medium term.
The average person isn't going to spend anything remotely comparable to what they spend on a car every year for an AI service. Even businesses have budgets now, huge demand doesn't matter if the willingness to pay isn't there.
The only path I can see to numbers like that is AI consuming existing business domains, even then it's very thin.
Say SaaS + legal + consulting + BPO + various other service industries collectively represent something like $10-20T globally.
Even if AI eventually replaces an enormous portion of that, it's probably not doing so at the same price. Why would customers switch otherwise?
Either the AI product has to be dramatically better, which is difficult for mature workflows, or dramatically cheaper which is much more plausible.
If it replaces $10-20T of existing services at roughly 1/10th or 1/100th (more likely) the cost, then you're looking at maybe a ~$1T AI revenue opportunity after replacing an absurdly large fraction of the existing service economy.
Who are now unemployed and can't pay for shit.
And that's before competition.
I think it's crazy to assume AI companies won't compete aggressively on price. As capabilities diffuse, smaller models catch up, inference hits pareto frontier the open-source alternatives have already improved and caught up, margins on routine intelligence should compress "hard" (emphasis on "hard").
We've already seen how difficult adoption can be even when the technology looks impressive on paper. Cheap here means 100x cheaper for 10x more demand that's a net 10x loss before any software or hardware optimizations.
So yes, I completely agree that cheap intelligence can bring an enormous amount of new usage.
"I just don't think usage means revenue." (you can plaster it on a wall if you want to, "usage doesn't mean revenue", if you want to find that out I have foss software bridge to sell)
The PC analogy actually reinforces this if you really think about it.
Compute became "vastly more useful" while the cost per unit of compute collapsed. Society captured enormous value, but all computer companies are literal failing giants without the AI hype. Value got caught by people who provided productionization.
Now if people expect AI to self productize itself I am happy to tell your try it. We all saw how OpenAI fell behind Anthropic because they thought that would work...
Google couldn't productize the search, instead they sold the eye balls and web-real-estate. Maybe that's the AI business model, but that's not $1T worth given you need to unglue people from other stuff.
Unless we get something approaching genuine ASI producing so much additional economic value that entirely new trillions, I don't see a path to $1-2T in direct AI revenue from customers.
The market simply can't absorb that level of spending.
Demand can be effectively infinite at the right price. But I think people are delusional on HN and SF if they think that number is in Trillions like the investments seem to suggest.
I am not saying Nvidia will fall tomorrow but someone will have to pull the breaks before this car goes to hell.
If AI compute is a transformative technology compared to industrialization (that's a huge "if", essentially positing a singularity-like outcome), that $1T-$2T/yr at current prices might be a tiny fraction of future GDP (real incomes), thus actually quite sustainable.
"The problem isn't demand it's, "how much people are willing to spend on it".
Lol its not even that - its what can I do with it? Which eventually has to show up somehow in the financials - from a macroeconomic stand point. Software production is microeconomic.
> The problem isn't demand it's, "how much people are willing to spend on it".
This is the right way to look at it, but a few of your estimates are a bit off. AI is being sold as an accelerator (or, if you're in a dystopian mood, total replacement) of knowledge workers. Currently knowledge worker salaries are $50 - 70 trillion a year globally, $10 - 11T in the US alone: https://gist.github.com/danielmiessler/2dc039762a202b083753b...
> Even if AI eventually replaces an enormous portion of that, it's probably not doing so at the same price. Why would customers switch otherwise?
AI is wayyyyyyy easier to wrangle than humans; no sick leaves, health insurance, perks, HR issues... heck they don't even sleep! If companies could replace us with robots, they would do so in a heartbeat. Capitalism!
So in a "what the market will bear" sense, we have an upper bound on the TAM. Indeed, I expect this is where Anthropic's ridiculous "$30 trillion" number is coming from... except now we see how they came to it.
If AI makes workers even 1% more efficient, that's a $500 - 700 billion value annually. In reality AI makes workers way more efficient (studies from the ancient era of 2024 showed about a 30% boost) so AI companies could realistically charge that much more. But then all the other factors you mentioned -- smaller models, competition, self-hosting, etc -- come into play, which put a downward pressure on revenues.
It's impossible to predict how these dynamics will play out, but the numbers involved are astronomical. This is why everyone from the frontier labs to Big Tech to VCs to nation states are scrambling to get in on it.
The typical bottleneck to wider batching on consumer hardware is memory capacity for the KV-cache, not compute (even unified memory/iGPU-based platforms have enough compute to allow for some batching, and SSD offloading changes the scenario entirely). Qwen models tend to have bulky KV-caches for any given token count. But agentic swarms might end up sharing a large cache prefix, so there's scope for potential gains there.
The raw margins on proprietary model inference are rumored to be quite high though (they have to successfully defray the entire investment into model training and datacenter capacity for inference, which is massive enough). The API cost you're paying for the model includes that raw margin.
> I believe this report has confused Opus (which is known to be around 5T) and Fable.
5T for Opus feels quite high though. DeepSeek V4 Pro is a mere 1.6T and often described as a match with Opus in overall quality. Even the largest open models in common use are around 2.8T.
Are these directly comparable, though? Doesn't Windows default to using free space on the system drive to grow the swap file in case of memory pressure? Linux has you set up swap files/partitions explicitly.
Also the way memory allocation in Windows works is that if it doesn't actually have the physical RAM (extended by pagefile) then the program will crash.
If you disable pagefile (like I did for some of my servers) and your program mallocs more than what you have in available space (even without ever dirtying a page) you will observe this.
I'm sure that this helps Windows a lot, even if it's not counted as actually used until the page is dirty.
Linux programs very often have virtual addresses many times the amount of physical ram, because there's never been any restriction. It's then very easy to just malloc huge chunks and use what you need and don't care about it too much.
Especially with many "tiny" allocations, python for example has huge sized objects which consume gobs of RAM dynamically, so any long running python process not only fragments memory but ends up having a bunch of objects consuming virtual memory...
idk why I felt the need to rant about this, but it's a difference that I've noted.
Yes, Windows doesn't overcommit memory. In the default config Windows is allowed to grow the page file as much as it wants, and empty pages are cheap to reserve in the page file (no need to actually write those zeros, you just need to reserve space for them). So the end effect is comparable, with the difference that software shouldn't just go ahead and ask for gigabytes of memory it doesn't actually need
I prefer the Windows approach, it's more predictable and has better behavior under memory pressure. But it can cause issues with software written under the assumption that the OS uses memory overcommit
If you have model layers/experts that reside in CPU RAM, it's generally better to do that part of inference on the CPU than pay the cost of shipping them over to the GPU. LLM inference is generally bottlenecked by data-movement, not compute. There may be a limited exception for prefill or perhaps decode of very wide batches, where shuffling the data around may be justified.
You're talking about multi-session vs. single-session throughput. A single user can easily leverage multiple sessions via e.g. subagent swarms, especially on a lower-end setup where any single session is going to be quite slow. Saturating utilization during off-hours is harder but potentially quite feasible by assigning lower priority, unattended tasks/inference loops.
Inherited wealth is not self-sustaining at all: if you spend it down, it disappears. Most inheritors of large windfalls will not manage to pass down that wealth in turn; it takes a rare ingrained cultural attitude to make it genuinely "self-sustaining", and even then that really means leaving the wealth as it is and managing it well - not using it to pursue any immediate "head start" in living standards over your peers.
The frontier labs will do well if they pivot their offering towards more capable, larger-scale models that are inherently harder to both train and deploy for commodity suppliers. Their existing investments in gigawatt-scale datacenters are quite optimal for this. "Commodity" inference need not comprise the whole market.
I don’t think this works, for a few reasons. First, intelligence gains from scaling the models bigger is sublinear now. So they could eke out a little extra performance, but the increased cost will eventually eclipse the economic value gained from this.
Second, humongous models are impractical even for them to deploy widely. They’re best used as teachers for smaller, more efficient models that can crank out the volume they need to sell.
Finally, there is a data wall. Sure, they can keep scaling RL on math problems and code. But with everything else, where will the supervision come from when they need several orders of magnitude more?
I agree. And even if they were able to do it for one more round, it's not a sustainable strategy. What they (Anthropic and OpenAI) need to do is build platforms and integrate verticals.
assuming the technology of model architectures does not gain any further breakthroughs that returns us back to the gains previously seen. I'm of the opinion that we still have some discoveries on the mathematical side of the fence to go that will improve models further.
> I'm of the opinion that we still have some discoveries on the mathematical side of the fence to go that will improve models further.
That's assuming the infrastructure needed to develop models stays available financially and supply wise. A lot of the services used to train and develop models are supplied and funded by people who are looking for multiple returns of investment. If/when OpenAI and Anthropic valuations fall and they inevitably get acquired, will Meta/Alphabet/Microsoft still want to spend lots of money for unclear returns in the short-term? Nvidia and co are on a one way train service to hype town. I don't think they will be happy to get on a coach to hype town Temu version. The shareholders likely won't.
Also, the backlash against LLMs is growing rapidly. AI content, data centres, etc is quickly gaining negative connotations outside of visual and music artists circles. While existing models are going nowhere, developing more advanced models is very quickly getting unpopular. LLMs Data centres increasing people's bills, Anthropic destroying old books, chat bots giving unethical advice to vulnerable people, etc. It won't be long before LLM infrastructure becoming an electoral issue.
Will a small research oriented community be big enough justify maintaining the apparatus needed to produce infra tech at a profitable level post OpenAI?
reply