On a related note, I was reading yesterday that apparently the real bottleneck for Chinese production of AI accelerators is HBM production, not processors or ASML equipment.
The lack of ASML EUV machines certainly hurts, and pushing DUV so hard results in abysmal yields of good chips, but you can compensate by running more wafers or making smaller chips, and the net result is that Huawei's Ascend production volume is limited by CXMT's HBM capacity not processor dies.
The problem is that HBM manufacture requires many steps (die thinning, via drilling, plating, alignment) where the equipment used by everyone else (Samsung, SK Hynix, Micron) is also blocked by sanctions, so the Chinese are having to develop all of this themselves too, which they have, but yields are currently low, even when using shorter HBM stacks.
andy_ppp 5 hours ago
Tokens per second is almost entirely memory bandwidth at inference time, training obviously needs more compute but you can add more chips for that.
cubefox 4 hours ago
According to SemiAnalysis, both inference and post-training (RLVR) is mostly memory bandwidth bound. Only pre-training is compute bound, but it now only takes a small share of overall data center capacity.
Not quite, it's got quite a bit more complicated with agentic use cases.
Prefill (input tokens) is heavily compute bound. And the ratio of input to output continues to rise, as typically in agentic sessions you have a few tokens output for a tool call and (many) thousands of input from the tool result.
Then you have cached input tokens, which is a totally different issue, system RAM or NVMe bound.
Obviously output tokens is VRAM memory bandwidth bound, but this is less and less of the bottleneck these days for overall agentic speed.
6510 5 hours ago
I read HBM yields are 25-30% (vs 80-90%) making them 3 to 5 times as expensive. They are 4-5 years behind, that probably means 1-2 in Chinese time.
throwaway85825 5 hours ago
Does that include yield from packaging?
vatsachak 5 hours ago
China should invest in an analog inference chip. It's a hail mary but why not.
geysersam 4 hours ago
They can probably afford to do both.
sroussey 4 hours ago
That would be like skipping land line phones for mobile...
clipsy 3 hours ago
Developing countries have done exactly that in many cases.
bell-cot 3 hours ago
I'd assume that they have - but will keep mum 'till they have a major breakthrough or large-scale operational deployment to announce.
danielheath 3 hours ago
The two challenges there are firstly - that analog design has been a separate electrical engineering school for most of a century, so there are few who could design it - and secondly - that every single chip will have subtle variations in its computations, necessitating some sort of model finetuning per chip. Possibly the chip could be characterised at the factory, and ship with the characterisation data burned into a controller rom or something, but if that doesn’t pan out the whole thing is likely a non-starter.
If it could be made to work, you could run a fable-grade model in tens of watts.
chvid 4 hours ago
Huawei uses their own non-standard HBM called HiZQ probably not produced by CXMT.
jacquesm 2 hours ago
The only thing the West is achieving here is that sooner or later China will be able to compete on their own terms rather than ours. It may buy some time but the end result is very predictable.
stingraycharles an hour ago
It baffles me how shortsighted the policymaking here is. Like, what did they expect to happen?
mrheosuper 34 minutes ago
they expect the Chinese to bend their knee and beg for the sweet, sweet chip
skeptic_ai 23 minutes ago
I think eventually The west will want to kneecap China, so I guess this is the best time to do so.
y1n0 4 hours ago
I’ve often wondered about die thinning. I mean just looking at sd cards and micro sd etc, it was clear that there had to be a thinning step (or many steps) and it seemed amazing to me that such a complex step could be economically viable. And yet it clearly is. All a matter of being able to operate at scale I guess.
It’s not a step I’ve seen discussed very much, so it was kind of neat to see it discussed front and center in a lay article like this.
fooker 8 hours ago
What's the main blocker (other than the current inflated cost) for using HBM instead of DRAM as the primary memory for consumer electronics?
refulgentis 8 hours ago
In one sense, nothing, in another, everything. It is DRAM, but the bandwidth requirements mean it’s paired to a processor, i.e. no DIMMs. Not 100% sure but things like MacBooks and the Framework tower, where you have fixed RAM for the device lifetime, have ~0 tradeoff.
nutjob2 8 hours ago
Nothing except CPU manufacturer choices. Mac laptops use it and they're consumer products.
People will have to get used to buying a fixed amount of RAM with their CPU but thats unlikely to be a problem.
chessgecko 8 hours ago
pretty sure its lpddr not hbm.
addaon 8 hours ago
> Mac laptops use it and they're consumer products
No, Mac laptops use LPDDR, currently LPDDR5X.
nomorewords 8 hours ago
The normal non-tech-savvy person already does this. They simply don't know that their ram is upgradeable or something else breaks first, before having to touch ram.
dylan604 6 hours ago
what is this too much RAM thing you mention? I thought the only valid RAM situation you could find yourself is not enough RAM. Too much? That's just fantasy
KeplerBoy 8 hours ago
MacBooks use regular soldered lpddr5(x) RAM. Same RAM as every other laptop manufacturer, they just use more lanes to achieve a higher bandwidth.
bunderbunder 7 hours ago
Perhaps more noteworthy for general home and business computing, doesn’t it also allow for lower latency?
Rohansi 7 hours ago
More channels or soldered memory? Channels are basically RAID 0 so it depends what you're measuring. Soldering memory down was the only way to use LPDDR5X so if you wanted the best memory you had to solder it down. LPCAMM2 exists though so newer devices can use that instead of soldering them down, but not all devices would be able to fit the required LPCAMM2 slots.
throwaway85825 5 hours ago
SOCAMM2 allows for removable ram in nearly 0 added space.
fooker 8 hours ago
Apple's "unified memory" marketing is so strong that even tech literate people seem to have this misconception!
Yup, the only reason Macs have higher memory bandwidth is because they use more memory channels, which gives them a wider bus. Both Intel and AMD only allow more than dual channel memory on server class processors these days.
kstrauser 7 hours ago
The “Apple only does X better because they do Y” thing has been a meme for ages. I remember dismissals like “PowerPC is only faster at math because it has more integer units” or something along those lines, and thinking, uh, isn’t that a good thing?
pixl97 7 hours ago
Depends on the expense trade off.
If I get 10% more performance for 50% more cost it really depends on one's needs, for example.
Rohansi 7 hours ago
I am just saying it's not magic and x86 is capable of doing the same. Quad channel memory used to be more common in consumer hardware but now it looks like you don't even have the option anymore for desktops. AMD's Strix Halo was the first sign to reversing that (it has quad channel memory!) and hopefully we see more of that in the future.
colejohnson66 6 hours ago
AMD does market segmentation and limits consumer chips to dual-channel. You need to cough up the dough and get Threadripper for quad-channel. Or even more for Threadripper Pro to get octa-channel.
Rohansi 5 hours ago
As I mentioned above AMD's Strix Halo has quad channel memory and is consumer level. But yes, other than that everything is segmented away.
torginus 5 hours ago
Afaik steam deck is quad channel, despite it being a pretty low end chip (with a decent GPU though)
Rohansi 3 hours ago
Kind of but not really. DDR5 splits your typical 64-bit channel into two 32-bit subchannels meaning the bus width is not increased. These subchannels are not always advertised because it's a just a feature of DDR5. Actually adding more channels increases bus width, which is what meaningfully improves memory bandwidth.
pdpi 6 hours ago
There's a qualitative difference between "they're doing a different thing" and "they're doing the same thing, tuned differently". GP is saying that this is a case of "they just tuned it differently".
This distinction doesn't change what the performance numbers look like today, but it does inform what changes would be necessary for those numbers to look different tomorrow. E.g. Apple Silicon isn't fundamentally orders of magnitude more efficient than x86, they just used smaller features. Newer Intel and AMD chips made on equivalent processes _also_ get similar efficiency gains.
sroussey 4 hours ago
There are AMD and Intel devices on similar process (not talking about A20Pro or M6 which are set to ship later this week), and they do not get the same gains.
And honestly, they have historically had different markets.
When the design is for only one customer, you don't need to generalize things, and those things you generalize to give different customers different options has costs.
AMD will soon be a larger customer for TSMC than Apple (NVIDIA is already there) so Apple's pre-booking new processes is likely to be gone in the near future.
JohnBooty 3 hours ago
"You're only better than me at sports because you practice more and try harder!"
Kon5ole 7 hours ago
The memory bandwidth is a small thing compared to the massive win you get by not having to move data between two memory pools at all.
Rohansi 7 hours ago
Depends on your workload. And AMD has supported unified memory long before Apple Silicon existed anyway.
fooker 6 hours ago
Often you don't move memory around as a programmer, but that's exactly what happens in the background.
It's the address space that's unified, not always the physical hardware.
The data movement (when needed) is handled transparently in the background by page faults and other tricks.
throwaway85825 5 hours ago
Except strix halo.
PunchyHamster 4 hours ago
Threadripper have that extra bandwidth and M5 still is faster
Dylan16807 an hour ago
Can you link a specific benchmark?
Keep in mind that non-Pro threadripper is still only 256 bits wide and Pro is 512. And the memory is 30% slower than with an M5. So an M5 Ultra has 3x the memory bandwidth of the best threadripper.
Kon5ole 7 hours ago
Having unified memory is a real advantage though, it's not a reality distortion.
davrosthedalek 7 hours ago
The price is that you essentially glue CPU and GPU together, which limits total compute, from a size and thermal perspective.
This is really not a limit because of unified memory -- in principle, PCIe GPUs could read/write main memory without the CPU. But it's a limit for /fast/ unified memory, because fast means close.
So unified memory is great as long as the integrated GPU is strong enough. Then it has two advantages:
a) probably faster transfer CPU<->GPU (but that's an implementation choice for the non-unified case
b) If you either need a lot of memory for the CPU or the GPU, but not for both at the same time, you pay for memory only once.
nvme0n1p1 7 hours ago
Agreed. That's why it's a good thing all computers made in the past 15 years have unified memory, not just macs.
Is that Marcan's vtuber persona (Asahi Lina) that changed name since Marcan doesn't work on Asahi Linux anymore?
Kon5ole 4 hours ago
In theory perhaps but the benefit is not as relevant with a weak iGPU. In practice all PC's with performance ambitions had a dGPU until Strix Halo and Panther Lake.
fooker 6 hours ago
It is a real advantage.
The reality distortion is that people seem to believe it's HBM, or somehow it gives you extraordinary amounts of vram. Neither are really true.
jorvi 4 hours ago
It isn't really, as long as you don't care about power consumption, physical constraints and money. Basically desktops <2025 (and hopefully >2027).
DDR is optimized for latency and stability at the cost of bandwidth whilst GDDR is optimized for bandwidth at the cost of latency and stability. GDDR is pushed so hard these days that a small percentage of errors is expected and corrected because this is still faster than running it slower but more accurate.
GDDR7 often has 10-20x the total bandwidth but 3x the latency of DDR5. Graphical workloads want as much bandwidth as possible but care relatively little for latency. Conversely, applications love low latency but don't really see any performance benefit from higher bandwidth.
So basicallyt you have workloads that are diametrically opposed and running unified memory forces you to compromise.
bob1029 8 hours ago
It's not that there's a blocker. It's that it takes roughly 3x the manufacturing capacity to produce an HBM package at the same storage capacity as DRAM. We are sacrificing total bytes for bandwidth.
tliltocatl 7 hours ago
How so? FEOL is pretty much the same, BEOL is almost the same save TSVs, the packaging tech is different and more advanced, but not exactly 1:1 comparable. Do TSVs really occupy 3x the area of DDR IO's?
monster_truck 7 hours ago
3x is a reasonable figure. They are not literally that large, though.
Yuck. Time to build an xSPI/HyperRAM workstation (if only these had multi-bank chips).
buildbot 6 hours ago
Sadly the $ per byte of xSPI and HyperRAM quite high
tliltocatl 6 hours ago
Yes, but enough to run a text editor, or even a mechanical CAD. Not an LLM, but that's the point!
Karliss 7 hours ago
It doesn't really say that it needs to use 3x more area, but that 3x more gets consumed due to "advanced packaging and manfuacturing complexity". Which doesn't properly explain why it consumes 3x more and could simply mean they have a bad yield and 2/3 produced is garbage.
tliltocatl 7 hours ago
Yea, that's the question. Yield situation can improve. Area overhead would not improve short of a completely new and incompatible tech.
bob1029 6 hours ago
> 2/3 produced is garbage.
This might not be far off the mark. You are irreversibly linking the fates of these devices after a certain stage of manufacturing. If something goes wrong at final packaging time, you lose all dies instead of one.
rkagerer 5 hours ago
I don't fully understand the source of the "total bytes" constraint, but a major factor may be because HBM4 / HBM4E can only make use of the footprint directly above the processor/logic die (or in direct vicinity of its interconnect), while traditional DRAM can be placed further away where there's lots of real estate on the motherboard.
I gather a practical max ceiling today is a stack of 16 chips in height yielding 64GB?
These chips have a massive bus size of 2048 bits, instead of the 64 or 128 bits (dual channel) used by DDR5. That's what gives them their order-of-magnitude bandwidth speedup. But even though they technically pack in more capacity per square millimeter of motherboard, I gather they take up more space than older technologies once you account for the vias and interconnects to route all those signals.
threecheese 5 hours ago
Thanks for that, just went down an interesting rabbit hole. Many of us were hoping this re-tooling would eventually trickle some fast RAM down to DRAM-exhausted PCs, but given it would require a rearchitecture of the motherboard it's unlikely.
sroussey 4 hours ago
HBM also trades bandwidth for latency, and your regular computing is much more sensitive to latency than bandwidth.
craigjb 2 hours ago
HBM4 has over 2048 signals to the processor’s PHY with tight signal integrity requirements that require the HBM stack to be < 0.5 mm from the processor die. That’s why HBM integration is done with interposers (soldered on the package). So, it’d be the CPU package that integrates it. Motherboard is too far away.
chessgecko 8 hours ago
I think people might prefer the lower idle power consumption from lpddr over the better bandwidth in hbm in battery powered stuff. That said right now the price is definitely preventing us from finding out.
fooker 8 hours ago
Idle yes, but HBM energy consumption / memory operations seems to be a bit better than DRAM.
vlovich123 7 hours ago
Only if you’re running at 100%. Consumers generally do not.
monster_truck 7 hours ago
That hasn't been true since early HBM2 days, before the controllers standardized on power/voltage management and did things like leave them in P0 to ship on time
vlovich123 5 hours ago
LPDDR/DDR/GDDR generally still win over HBM when there’s no data being transferred. HBM is primarily better in watts/byte transferred. Consumer electronics spend most of their time idle.
Zagitta 5 hours ago
Racing to idle is a very common power optimization technique
vlovich123 4 hours ago
Right, but HBM idle is significantly worse than LPDDR idle or even DDR idle for that matter. That matters a lot precisely because the device is idle most of the time. Your idle power draw dominates.
phkahler 8 hours ago
>> What's the main blocker (other than the current inflated cost) for using HBM instead of DRAM as the primary memory for consumer electronics?
HBM is meant to be integrated into the same package as the CPU, so no more DIMM sockets. It also has higher latency apparently.
reliabilityguy 8 hours ago
HBM is a stack of DRAMs, so there is no “instead”.
buckle8017 7 hours ago
The vias to enable stacking is a significant amount of the die area.
saltcured 7 hours ago
If they're talking about production capacity, that is some product of die area and process steps, right? It doesn't have to be 3x die area, just 3x lower factory throughput for the same number of functioning memory bits.
buckle8017 3 hours ago
HBM is less dense at a water scale than DDR because of all the vias, but each memory but is as dense or denser.
You make HBM instead of DDR and the number of bits you're making goes down.
It's really that simple.
HarHarVeryFunny 6 hours ago
Why would you want/need to?
The advantage of HBM over regular non-stacked DRAM is memory bandwidth, which also requires a super-wide memory bus - 2048 bits wide for HBM4. Compare that to the 128 bit wide bus of a modern CPU.
So to take advantage of it on the desktop, or anywhere else, you need that 2048 bit wide bus, and a processor capable of consuming 2-3 TB of data per second!
These are not normal requirements, other than for a GPU.
fooker 6 hours ago
SIMD (and especially the modern matrix extensions) can use as much bandwidth you can throw at it.
b112 6 hours ago
Without further clarification, that statement seems impossible. "As much" being unbounded and all. You should expand what you mean.
fooker 5 hours ago
This will blow your mind, but it actually is pretty close to being unbounded. :)
Consider the 'MMA N matrices' primitive modern CPUs are starting to support. For the current generation of CPUs, N is a constant like 16 or 32, but there's nothing preventing it from being 1024 or larger if we have more memory bandwidth.
All this with a single instruction.
articulatepang 3 hours ago
Surely something prevents it being 1 quadrillion bits per instruction? Since that’s well within “unbounded”.
XorNot 6 hours ago
Right but if HBM memory is all that people want to produce, then building a CPU which can use it use it would be useful on it's own merits.
But in reality we also already have unified memory architecture systems, integrated graphics etc.
p1esk 6 hours ago
People want to produce hbm because it’s more expensive and more profitable than regular memory.
XorNot 5 hours ago
There is a world where scale and experience means it's about the same though, is the thing.
And memory is already expensive. It's downright hard to even get it though - you frequently would prefer not what's cheapest, but whatever is in largest scale production.
Dylan16807 an hour ago
Scale and experience almost entirely share between HBM and normal memory. And they're both in large-enough scale production to not have a big difference on availability; if you're willing to pay HBM prices you should find even more sellers of DDR.
The only way I see HBM becoming competitive for consumer CPUs is if they solve the yield issues. Or if AI crashes so hard that people are putting those GPUs on fire sale and salvaging mass quantities of HBM off of them.
Dylan16807 2 hours ago
> Compare that to the 128 bit wide bus of a modern CPU.
Or 256-512 bits on medium to high end consumer CPUs if you're apple.
At least DDR6 is probably widening things 50%.
torginus 5 hours ago
Mainly bus width. Afaik HBM is like 1024 bits vs DDRs 64 so you need lots of transfers in parallel to saturate the bus, and CPUs kinda want 64 bytes of data as that's the size of a cache line ASAP. So you need a ton of in flight transfers which isn't a thing CPUs provide, maybe multicore workloads.
Buy the way you win with CPUs is with latency, and not bandwidth, which is why Apple M series actually uses DDR with lower latency because of the stacking.
gs17 8 hours ago
A shame that this should if anything, lead to consumer DRAM prices getting even worse.
nicoburns 8 hours ago
Why would more RAM supply lead to higher consumer prices?
kenny11 8 hours ago
If Samsung is limited in the number of wafers they can process per month and they use more of those to produce HBM they necessarily have less of them left to make other products, like consumer DRAM.
deagle50 8 hours ago
Presumably DRAM production capacity gets reallocated to HBM, not sure.
shevy-java 8 hours ago
Depends on how quickly they are sold.
zargon 8 hours ago
Memory allocated for HBM is memory taken away from DDR production.
matja 8 hours ago
Depends on what proportion of Samsung's output is current HBM4 and HBM4E DRAM.
Everything in their statement can be true and it be a bad thing for consumers of non-HBM RAM.
"HBM capacity to expand to 250,000 wafers a month"
So let's say their current HBM capacity is 100k wafers/month (pure speculation/random number for illustration), and their total RAM capacity (including HBM( is 300k wafers/month, then non-HBM capacity reduces from 200k to 50k.
EgregiousCube 8 hours ago
Keep in mind that total manufacturing capacity is increasing as well. Perhaps not enough to fully offset, but it's incorrect to assume that supply capacity is flat.
pixl97 7 hours ago
Question is what does the demand curve for regular ram look like also? If regular production is on a 10% increase curve but a 40% demand curve than we've been at the samw place.
cogman10 5 hours ago
It'll take a lot of time.
Micron has been working on expanding in Boise since around 2023. They are predicting that the first new chips will start rolling out around 2027.
I don't think we'll see any relief until at least 2028 or even later. And even then, probably not without antitrust laws being enforced. Because you know memory manufacturers won't just simply drop prices, they have a long history of price fixing.
gs17 8 hours ago
And it's worse than that, Micron said that converting capacity to HBM was at a 3:1 ratio, so that 250k HBM could be up to 750k in non-HBM. Fortunately, some of the increase is due to improved processes.
no-name-here an hour ago
OP article estimates HBM is going from 40% to 80% of their output.
devy 8 hours ago
HBM4 and HBM4E DRAM are NOT the DDR4/5 that consumer markets need. Capacity allocation is leaning more to data center grade HBMs so less to produce dedicated DDR4/5 DRAMs. Supply demand will further drive up the consumer DRAM price! Note, the article mentions NO of new fabs is being constructed (all semiconductor manufacturers know that constructing more fabs means the boom/burst cycle will eventually kill them, so no one create more fabs) Perhaps the federal government need to step in here - the market doesn't fit the issue.
tipsytoad 7 hours ago
Plus it takes 3x the wafer capacity for hbm than the same byte capacity in dram, so we’ll likely see the consumer market be decimated here
chr15m 3 hours ago
The consumer market will not be decimated. Memory of all kinds will get very cheap quite soon because of supply and demand and substitution.
sgt 7 hours ago
Might lower the prices of Mac Minis, Studios etc though
airspresso 7 hours ago
No, those use LPDDR5(x), not HBM.
bnjms 6 hours ago
HBM can be used for GPU though. I wish it was more common.
chr15m 3 hours ago
No, completely wrong. Supply and demand will definitely fix this.
More HBM4 will drive the price of it down. That will make it less profitable to produce. That will mean firms shift some production back to more profitable "consumer" RAM. Increased supply will lead to lower prices.
A shortage is always followed by a glut.
no-name-here an hour ago
> That will mean firms shift some production back to more profitable "consumer" RAM. Increased supply will lead to lower prices.
Long-term, sure. But the article we are discussing is about next year, non-HBM supply going from 60% of their output to 20% (HBM 40%→80%).
chr15m 8 minutes ago
> Long-term, sure.
Yes. Non-HBM Samsung supply going from 60% to 20% will cause prices to increase, which will cause increased production of non-HBM, and then prices will fall (below where they currently are).
Dylan16807 an hour ago
What specifically did that say that you think is wrong?
Supply is currently moving the wrong way. Supply and demand for consumer DRAM will get even worse, even if it will "definitely fix this" in the long run.
And the fact that production is moving the wrong way suggests that in the short term supply and demand is actually doing more harm than good. HBM is so profitable it's radioactive.
I don't know where you got the idea in another comment that all memory will get "very cheap quite soon". They're not increasing total memory production. (much/yet)
chr15m 15 minutes ago
> What specifically did that say that you think is wrong?
You said:
> the market doesn't fit the issue.
I presume you meant "fix" not "fit". You are wrong that the market will not fix the issue. There are literally millions of observed cases across many markets of increased demand (or decreased supply) leading to higher prices leading to increased supply leading to lower prices, which is why it's a fundamental law of economics.
The price of RAM will fall.
stevefan1999 8 hours ago
Gamma squeezed by the AI business, that's why. The more you add the fuel, the further it will pump up.
mattstir 7 hours ago
Part of the implication is that factories that could be producing consumer-facing DRAM like DDR5 would be retooled to produce HBM instead, leading to even less total consumer RAM production.
m4rtink 3 hours ago
The crash won't be kind to them & they deserve everything they will got for their past behavior.
robotnikman 5 hours ago
Maybe this leads to device manufacturers using HBM memory instead.
amelius 8 hours ago
Will that be enough for AI's hunger?
heaney-555 8 hours ago
Not even close.
gs17 8 hours ago
Samsung is already building another fab (which was originally suspended due to low memory prices years ago). Hopefully when it goes live in a few years it's not all dedicated to HBM for AI.
GoToRO 8 hours ago
It will be just in time for when AI will run very well on consumer hardware and the need for data centers will collapse.
EA-3167 8 hours ago
Or even more amusing in time for the bubble to burst, I hope the big RAM makers end up holding the whole bag for that. Greedy bastards.
bethekidyouwant 8 hours ago
Since you’re not greedy when there is a memory glut I’m sure you’ll be willing to pay extra to make it fair.
EA-3167 7 hours ago
Not at all, I’m going to relax with a cooling drink and watch the consequences of this unbelievably destructive and wasteful venture implode. I also adore the idea that it's on the customer to be "fair" to a company that dumped us a group in favor of chasing B2B money.
Their choice, their consequences.
pixl97 7 hours ago
Internet is a fad, it will implode at any moment.
EA-3167 7 hours ago
In fact as a business opportunity it did just that because the massive investment in the mid-late 1990's was without anything like a connection to profitability or sustainable demand. It took years after that crash for the concept of e-commerce to begin both a recovery and evolution into what we see today.
I expect something similar for AI. It's useful tech... just not very profitable tech, and certainly not to the tune of trillions of dollars worth of public demand. The promises of superintelligence, replacing everyone, and the rest of the hype will die with the companies who made the promises, but the tech will survive and thrive.
pixl97 7 hours ago
>The promises of superintelligence, replacing everyone, and the rest of the hype will die with the companies who made the promises
The hype will die but there are many reasons why super intelligence and robots are a separate entity from said hype.
Look, back a few decades and tell someone our gdp would be in the trillions and it's likely they'd have a hard time believing you. A huge portion of our products that we use day to day would be complete science fiction to them.
All we're negotiating at this point is the timescale it will take.
card_zero 6 hours ago
Unpredictable things happen routinely, therefore these specific unpredictable things will happen?
pixl97 5 hours ago
These things aren't really unpredictable. Work long enough on a robot with more dexterity and you will get just that. The timetable on when it happens is a bit more up in the air, economic downturns and wars can have huge impacts on it.
But in my mind robots and superintelligence are inevitable unless there is a massive setback in humanity before then. Nature already did it once. We're not inventing something totally new. Add in every new invention and bit of intelligence we build on and actualize pushes us that much closer to the goal. Information technology allows this to speed up even further with easy sharing of information and testing.
Superintelligence is not like faster than light travel. We have many frameworks that show us FTL is impossible. I don't believe there is a single widely accepted framework that shows there is some limit to intelligence and we're near it. Intelligence isn't even a singular thing. My calculator is a super intelligent adder compared to me. It would seem highly improbable that somehow nature random walked into making brains the most efficient general intelligence device in all dimensions and scales.
usef- 4 hours ago
I think they're talking about the past: memory has often been boom/bust. Memory manufacturers lose money during gluts, and did so long before AI. People benefited from cheap prices at that time.
glub 5 hours ago
Nearly the entire reason we're in this mess is because RAM/SSD/HDD makers are terrified of AI bubble bursting.
Long-term contracts, not building new fabs - they're in full on hedging mode right now.
leoc 3 hours ago
Seemingly the AI bulls are either not sufficiently optimistic, or not sufficiently wealthy(?!), to build new fabs as joint ventures with the manufacturers in which they agree to assume most of the downside risk?
no-name-here an hour ago
They signed long term contracts for the hardware delivery over years. AI investment is approaching 1 trillion per year and expected to grow to well over 1 trillion per year. [1]
Even over their many years, projects like the Manhattan Project, the Apollo Program, or the U.S. Interstate Highway System never added up to that. [2]
Is your argument that there is insufficient AI spending at present as they aren’t also taking on building their own fabs?
If the long term contact are with any AI companies, then good luck getting anything back in bankrupcy proceedings!
m4rtink 2 hours ago
Looking forward to see them burn.
IshKebab 8 hours ago
That's never going to happen. By the time you can run current frontier models on your $10k desktop the frontier will have massively advanced and people will want those models instead.
kingleopold 7 hours ago
This + your ROI in $10k device will be always lower than busy datacenter, its literally math. They sell free compute to others when you dont use it, you will never sell at that level or even you magically sell home compute, you will not compete at price
leoc 3 hours ago
That greater efficiency only benefits the LLM SaaS providers as long as the hardware manufacturers, probably especially the VRAM manufacturers, remain supply constrained, since the high-efficiency users are the ones who can pay top dollar. But the hardware guys' dream is presumably to get parts into millions of laptops which remain on standby for 19 hours a day, not to bargain with SaaS providers who obsessively optimise their memory consumption.
hypfer 7 hours ago
I'm not sure if this prediction will hold true.
We're not seeing the progress in those "frontier models" that we have previously seen. There's certainly still gas left in tank tank, but we're way into the diminishing returns by now.
Cloud inference still beats hardware investments by orders of magnitude of course, but that's only if your data doesn't really matter to you.
airspresso 7 hours ago
We are certainly not in the diminishing returns phase for LLM progress. No sign of that yet.
hypfer 7 hours ago
Well I mean if I wanted to be extra pedantic, I would argue that we've been in that phase since LLMs were first introduced.
Before that, we had 0.
After that, we had more than 1.
A leap as far as that is hard to recreate.
But that wasn't my point. That's just trolling.
The actual point is that LLMs aren't gaining new capabilities anymore. They just get more reliable at the ones they already have; turning what was a coin flip to some higher probability.
That's (intuitively speaking, not strictly mathematically speaking) kinda the mathematical definition of diminishing returns.
bunderbunder 7 hours ago
I’ll grant that for specialized applications like coding agents and mathematics, but even there I suspect that most the real gains are actually taking place in the harness.
But I suspect returns may have already diminished into negative territory for at least some other use cases. One of my least favorite job responsibilities in this brave new era is figuring out how to avoid performance and behavior regressions when an older model were using for some application reaches end of life. It’s getting uncommon for me to look at our benchmark results and say, “Oh, good, it does better on one of the newer models!”
pixl97 7 hours ago
>suspect that most the real gains are actually taking place in the harness.
Part of the reason harnesses work well is you can run a lot of agents in parallel. That doesn't slow down demand.
hypfer 7 hours ago
That is true, but the eventual realization that more machines doing more coin flips in parallel does not mean "more work gets done" might.
LLMs are amazing tech, but they're terrible without oversight. More agents faster just makes reality collapse on them quicker.
But yeah, you're right, temporarily, this will still push demand. But the topic was about "diminishing returns" as in "tech getting better". Not as in "customer spending".
pixl97 7 hours ago
It's kind of weird because more machines working together does mean more work gets done. Coin flips and weighted coin flips are totally different things. Any biases weights towards reality push you closer to reality when you use them.
New models keep being able to use more and more agents on longer time frames. Your hypothesis doesn't look like what we're measuring.
hypfer 7 hours ago
Who is we?
pixl97 6 hours ago
The people mapping AI capabilities.
hypfer 6 hours ago
Oh cool, so that we includes me! :)
pixl97 5 hours ago
Maybe turn on your light when you use a ruler? Not sure what else to say.
bunderbunder 7 hours ago
I had actually been thinking more about all the non-LLM functionality that go into the harnesses. I'm not going to name names and I haven't done any rigorous testing, but my general impression is that choice of harness matters more than choice of model. In terms of basic task completion success specifically, not code aesthetics.
pixl97 6 hours ago
A perfect harness will not extract gold from a dumb model. It's a system that builds on each other, though we've not probed that frontier much to have a good intuition on what effects what.
leoc 2 hours ago
But high demand for LLM time isn't sufficient to keep customers at the frontier LLM SaaS providers. That demand can be satisfied locally or at non-frontier outlets, absent hardware shortages at least. The Tier 1 providers (and the would-be Tier 1s) presumably need to open up a much bigger lead in model quality, one that doesn't simply get distilled away this time, and/or continue to be protected by ongoing (or worsening!) hardware shortages. (And that's overlooking the revenue shortfalls which OpenAI and Anthropic seem to be facing already.)
3eb7988a1663 6 hours ago
One thing that I really want to know - the better models from today vs a year ago - what has changed. They have already pre-trained on all available public data. Scooping up the last percentage of archaic texts which were never digitized is not going to move the needle.
Is it just that the providers are generating tons of synthetic datasets on coding tasks so that the models get more exposure to the right thing to do? Every time someone points out an LLM stupidity they add some training data to patch over the weakness (trivial to generate "there are two 'l's in llama")?
48488448 7 hours ago
they really dont want to hear this bro lol
hypfer 7 hours ago
I can see that by those reddit-style vote swings, but who are "they", exactly?
Who is so emotionally invested into random comment sections being purely positive about their pet.. uuuuuuuh.. tech?
Very weird.
gehsty 7 hours ago
It’s a constant tension in computing that has been around since mainframes and clients… Neither is going to disappear. My general feeling is normal people care more about how thin and light something is than their privacy, so if data center powered LLMs will have a strong future.
hypfer 7 hours ago
Hmm I'm not 100% sure about that, given that edge is very viable, and the geopolitical climate has changed quite significantly.
I agree that datacenters are not going to go away, but I have doubts that the buildup that has happened is really going to pay off for most operators.
bunderbunder 7 hours ago
I’m not so sure about that. Already AI vendors are back to cutting prices to try and keep customers from cutting back on their usage. My own employer is working hard at pivoting to much smaller fine-tuned models for established use cases, and seeing model performance improvement in addition to large inference cost reductions. Being able to run them locally hasn’t exactly been a disaster for devex, either.
It may turn out that demand for SOTA frontier models isn’t so limitless after all.
pixl97 7 hours ago
Every product follows demand curves. At a price of 0 you could find infinite usage. This has nearly zero relation to how much it costs to provide the product.
bunderbunder 6 hours ago
Except of course it relates. All else being equal, we will prefer $X COGS over $2X COGS because that helps us with both profit margins and price competition.
pixl97 6 hours ago
It relates in the sense there's a minimum cost of production without losses, not the actual price people are willing to pay.
bunderbunder 5 hours ago
Framing it in terms of the price people might be willing to pay for a single product in isolation frames the point I was making, which was about price competition, right out of the picture.
Maybe I'd be willing to pay $10 for product A if I had other options. But if there's a product B for $3 that's not quite as nice but still ticks all my boxes, then product instantly becomes a lot less attractive.
blurbleblurble 7 hours ago
It's going to happen very soon, which is why these frontier labs are scrambling to shut down open source language models. There's an existential risk threatening their obscene returns.
IshKebab 6 hours ago
> It's going to happen very soon
Why? You can't just assert it. There are very good reasons to think it won't happen soon, and you've given no reasons to think it will happen soon.
dwedge 6 hours ago
You asserted that it was never going to happen first
geysersam 4 hours ago
But he gave a reason for that. "Before that happens the frontier will move". Why do you think it will happen anyway? Do you think the frontier will not move fast enough that local models are unable to catch up, or do you think people will prefer local models at a point. Or something else?
blurbleblurble 4 hours ago
Because everything is converging on a backlog of huge efficiency gains established in research, waiting to be combined. Looped transformers, a whole host of diffusion techniques and new quantization techniques, maturation of ternary distillation and new ways to separate logic from stuff that can be looked up. It would surprise me if most frontier models were actually even that big at that point in terms of active params. I highly doubt it.
mrlonglong 6 hours ago
It is for that reason they are being archived and torrented as a very large middle finger.
usef- 4 hours ago
I feel like cost competitiveness of local has been going down, if anything, not up. API providers can use hardware more and have scale efficiencies. Do you see any reason this will reverse?
dabinat an hour ago
You don’t need frontier-level performance for every task. That’s why companies hire both junior and senior developers. I suspect a decent percentage of people using Fable would probably be fine with Opus.
Also, the frontier can’t keep advancing at this rate forever. Eventually the low-hanging fruit will all be gone and advances will slow down.
konaraddi 7 hours ago
There will be demand for both
amelius 6 hours ago
What if the AI companies start selling the hardware with their models?
I.e., you get a locked down device with access to their AI and maybe a way to run third party apps. Of course the developers of those apps will have to pay a percentage of their revenue.
Sound familiar?
throwaway85825 5 hours ago
More likely they would sell a chip with the model etched in silicon for the dramatically higher token/s.
amelius 4 hours ago
A chip that can only be accessed through their software. With a monthly subscription fee, and other enshittification surprises.
szatkus 4 hours ago
Most likely those models would be extracted from the hardware in no time.
amelius 4 hours ago
I suspect no, because most of the tokens would be used for its internal reasoning loop, and a cheaper model could be used to obfuscate the output.
nutjob2 8 hours ago
Not until the industry gets severe indigestion, which doesn't seem that far off.
mixedbit 7 hours ago
During the Covid global chip shortage Intel announced new factories to address the production bottleneck, before the factories even started to be constructed, the shortage was long over and the projects were eventually canceled.
glub 5 hours ago
tl;dr; This is existing production being redirected to HBM. Which means consumer market is going to get a lot worse.
And the worst thing is that is the best possible strategy for them. It's essentially win-win for everyone but consumers.
- Buyer (AI) has crazy money, so will pay whatever
- Seller doesn't have to build anything new, as the buyer is willing to pay whatever
- Generate ridiculous profits from crazy money
- No oversupply risk in case of reversal
- Return from producing HBM to DDR5 in a single quarter if reversal does happen.
Hold on to your existing hardware, people, and be on lookout for your local deals.
radiator 4 hours ago
The solution would seem to be preventing that "Buyer (AI) has crazy money". They already have borrowed billions and are only generating losses so the money should run out. It just needs to be made sure that nobody (taxpayers) will rescue them.
yonaguska 3 hours ago
> just needs to be made sure that nobody (taxpayers) will rescue them.
oof.
m4rtink 2 hours ago
This insane nonsence needs to stop.
chr15m 3 hours ago
A shortage is almost always followed by a glut. Memory of all kinds is going to get very cheap due to supply and demand and substitution effects.
cute_boi 3 hours ago
Not when companies have monopolies.
chr15m 3 hours ago
There is no monopoly on computer memory.
Yes, only a small number of companies produce memory because of capital requirements and complexity. When prices go up both of those things become less of a problem for investors who can see better return for lower risk on that high capital investment. They will then invest in production to capture those returns. This is literally the exact reason Samsung are doubling supply of this particular type of memory.
High prices will fix the shortage.
ajnin 43 minutes ago
Memory factories cost billions to tens of billions, "investors" aren't going to spend that kind of money if they are not reasonably certain that they are going to make a profit, and the market is very uncertain right now with everybody bracing for a bubble burst at any moment. It's not the first time there has been such a shortage in the computer parts market, and the oligopoly, be it RAM or hard drives manufacturers, has always managed to keep prices high and production low enough to avoid a price crash after recovery.
chr15m 23 minutes ago
> hard drives manufacturers, has always managed to keep prices high
No. Please take a look at the unit price for hard drive storage 10 years ago as a percentage of what it is today.
Taniwha 4 hours ago
You just know that the AI bubble is going to break and they're going to be stuck with tons of HBM4s.
The big problem is that this will not depress the consumer DRAM market and mean we can all afford DRAM again
blurbleblurble 7 hours ago
The beginning of the bubble pop?
esskay 7 hours ago
its ram for the types of systems ai needs in datacenters so...no?
blovescoffee 6 hours ago
how would you even derive that conclusion from the headline/article?
badgersnake 7 hours ago
Now make some DDR5.
pixl97 7 hours ago
HBM is DDR5 in a different package.
badgersnake 6 hours ago
If it doesn’t fit it’s no use to me.
shevy-java 8 hours ago
About 2 years ago I bought some SDRAM or something like that, DDR4 or DDR5, don't recall offhand. A few months ago I looked at the price today, and it was over
3x as high. That's just insane. Governments need to do something against this abuse system that AI amplified here.
pfdietz 8 hours ago
What abuse? Responding to supply and demand fluctuations isn't abuse, it's the proper operation of the market.
dumberquestions 8 hours ago
Would if one industry gobbles up an important resource to the point that other consumers lose practical access to it, shouldn't there be limits?
jefftk 8 hours ago
No. The other industries can bid for access to the resource, and under most circumstances, whether they are willing to bid higher tracks importance. Sometimes there are cases where something has significant positive externalities and so people are not willing to bid high enough, and then we can talk about some kind of government intervention (typically a subsidy that attempts to track the value of the externalities) but I don't see that applying here.
pfdietz 8 hours ago
If others are willing to pay more, that a signal that more value is created if the resource goes to them rather than to the more price sensitive customers.
applfanboysbgon 8 hours ago
This logic falls apart completely when gamblers allocate one trillion to bidding up the price, denying access to the resource to people who are responsibly spending their own money. The ideal world of the imaginary perfect free market never takes into consideration the messiness of the real world, like the fact that people will spend extreme sums of money irrationally. It can take years for this effect to correct, damaging the market severely in the meantime.
Alternatively, the money invested can be rational because it prices out competitors and establishes a monopoly, after which point the monopolist earns their absurd investment back with complete control of the market. This is also bad.
pfdietz 7 hours ago
You can reach any conclusion you want if you assume others are behaving irrationally. But why should I assume they are wrong instead of you being wrong?
pixl97 7 hours ago
This has zero to do with the company that is making the product and all to do with the company buying them.
The problem you have here is hundreds of different companies are doing the "gambling" in a non collusionary manner so it's going to take decades in court to prove it. Are you saying governments should do an authoritarian take over of RAM allotment?
pfdietz 6 hours ago
And even if the companies are "gambling" (and what company doesn't do that) it doesn't mean the gambling is irrational.
joshheitzman 7 hours ago
Where is the additional supply resulting from the sustained increase in demand?
> Governments need to do something against this abuse system that AI amplified here.
Which governments, and what exactly would you want these governments to do? The demand is global. There are no levers a single government can pull to meaningfully influence the global demand without fully committing to an protectionist economic policy, in which case the U.S. doesn't have the facilities to magically pop up world class fabs overnight, and South Korea and Taiwan don't have the market demand that the U.S. generates to justify their investments in making these chips and China lacks the IP to be able to build anything comparable to Nvidia's silicon at the moment.
No one has all the cards and no one controls all the levers.
carlm42 8 hours ago
Governments have sued DRAM companies for price fixing and forming cartels before: https://en.wikipedia.org/wiki/DRAM_industry_price_fixing. They can absolutely step in and if there are evidence of price fixing or anti-competitive behaviours, can impose fines. The choice of whether to do that though is political.
daedrdev 7 hours ago
Several of these companies almost went under before the AI boom. There isn’t a cartel.
lovich 7 hours ago
Is there any allegations of price fixing in this case? The demand just skyrocketed. Other than legislation like the defense act in the US I don’t know of any governments that have jurisdiction over these ram makers that mandate companies produce more of a product.
Especially not for consumer goods.
wmf 7 hours ago
If they're charging one customer $X and another customer $5X for the same chips... there's got to be some law about this.
srdjanr 5 hours ago
I missed that this is happening, is it really? I thought it's 5X for everyone
wmf 5 hours ago
Supposedly OpenAI is paying 2025 prices.
CamperBob2 8 hours ago
No one has all the cards and no one controls all the levers.
I'm as fanatical a free-market fundamentalist as you'll ever meet, but if someone were to argue that government has a role in preventing bullshit like OpenAI's unilateral 40% attack on the entire DRAM market, backed by nothing but funny money, I would have a hard time coming up with defensible counterarguments.
bunderbunder 8 hours ago
I don’t really know much about how this stuff works, but I have a feeling we wouldn’t even have to do anything punitive. We might just need to find and close whatever weird loophole allows the folks who are participating in this gold rush to feel confident enough that they’ve externalized their risks to be willing to engage in speculative data center buildout projects on such a grand scale in the first place.
pfdietz 7 hours ago
Your attempts to insult what they are doing doesn't show they did anything wrong, but it does impeach the validity of your own argument.
CamperBob2 7 hours ago
Hit dog hollers, sounds like.
pfdietz 6 hours ago
That's abuser logic.
CamperBob2 6 hours ago
The funny thing is, we're mostly in violent agreement, going by your other comments. This is a weird point to disagree on... so weird, in fact, that it's easy to believe you are associated with OpenAI. Hence my follow-up comment.
What Altman did, and the way he did it, was not OK. Unless you believe that humanity's adoption of AI-based intellectual work is best served by concentrating an enormous amount of power in one company at the expense of everyone else, turning the market into a literal zero-sum game. Do you?
rangestransform 6 hours ago
Apple used to do the same thing with TSMC’s newer process nodes, and nobody really complained because people were getting superior products. I happen to like US antitrust enforcement that considers consumer benefit primarily.
newsclues 8 hours ago
How much memory does it take for the consumer market not to be totally screwed?
Can that share of production be allocated to consumers, and the AI fights over the rest?
m4rtink 2 hours ago
Well, thats kinda how strategic reserves work - so that say food manufacturers just don't dump all the cheap food to sell crack instead & everybody starves to death.
m4rtink 2 hours ago
The same governments that dumped bilions into to these companies and gave them insane tax breaks.
rtpg 2 hours ago
The same major memory makers got prosecuted for cartel behavior by the DoJ a little over 20 years ago[0].
One might say "well what specific law have they broken" and I don't know and I don't know if they have, but at the very least the power of the State _could_ be used to force behavioral changes in some ways or another.
