LPDDRX is used in some inference hardware. The same stuff you find in laptops and smartphones.
Also, the servers need a whole lot of regular CPU DIMMs since they’re still mostly EPYC/Xeon severs with 8 GPUs in each. And why are they “wasting” so much RAM on CPU RAM that isn’t really needed, you ask? Same reason as a lot of AI: it’s immediately accessible, already targeted by devs, and AI dev is way more conservative and wasteful than you’d think.
Same for SSDs. Regular old servers (including AI servers) need it too. In a perfect world they’d use centralized storage for images/weights with near-“diskless” inference/training servers. Some AI servers do this, but most don’t.
Basically, the waste is tremendous, for the same reason they use cheap gas generators on-site: it’s faster-to-market.
Hardly. Power costs are trivial to them at the moment, and a server hardware bottleneck would just consolidate power to the big few that can afford it (which is what they want).
To add to what others said:
LPDDRX is used in some inference hardware. The same stuff you find in laptops and smartphones.
Also, the servers need a whole lot of regular CPU DIMMs since they’re still mostly EPYC/Xeon severs with 8 GPUs in each. And why are they “wasting” so much RAM on CPU RAM that isn’t really needed, you ask? Same reason as a lot of AI: it’s immediately accessible, already targeted by devs, and AI dev is way more conservative and wasteful than you’d think.
Same for SSDs. Regular old servers (including AI servers) need it too. In a perfect world they’d use centralized storage for images/weights with near-“diskless” inference/training servers. Some AI servers do this, but most don’t.
Basically, the waste is tremendous, for the same reason they use cheap gas generators on-site: it’s faster-to-market.
It only just now occurred to me how much the war in Iran is also fucking over AI companies.
Hardly. Power costs are trivial to them at the moment, and a server hardware bottleneck would just consolidate power to the big few that can afford it (which is what they want).
Happy giggles.