Some great stuff so far at Hot Chips…Takeaways from the key talks below…
Feeding AI’s voracious hunger for data
Speaker: Jim Handy, Objective Analysis
Summary: Handy framed the memory cycle around hyperscaler AI spending, HBM’s heavy wafer requirements and limited spare DRAM capacity. He also covered NAND/HDD spillover, pricing, efficiency gains and processing closer to memory.
1. Hyperscaler capex remains the core demand driver
Handy said the biggest force in memory is the sharp acceleration in hyperscaler AI capex. Spending had historically grown steadily before breaking higher as the major platforms began competing to build AI infrastructure. That flows directly into memory because nearly every major AI accelerator, including NVDA GPUs, GOOGL TPUs and custom silicon, uses HBM. Even some networking components now use HBM, creating a demand category that barely existed several years ago.
2. HBM is creating a much broader DRAM supply problem
HBM produces roughly one-third as many gigabytes per wafer as standard DDR because of its larger die size, TSVs and supporting circuitry. That means rapidly rising HBM demand consumes enormous wafer capacity. DRAM suppliers had also gone more than a decade without major capacity additions because process shrinks were enough to satisfy historical bit growth. New fabs can take around two years even under aggressive timelines, leaving the industry unable to respond quickly to the current demand shock.
3. Tight supply is flowing directly into pricing and memory-vendor revenue
DRAM and NAND are commodities, so shortages let suppliers shift capacity toward customers willing to pay more. Handy noted that spot prices have risen roughly 7x from prior levels, though spot is an imperfect industry indicator. Both DRAM and NAND price per gigabyte have increased sharply, driving unusually strong revenue growth at Samsung, SK hynix, MU and major NAND suppliers.
4. AI infrastructure is also tightening NAND, SSD and HDD markets
META tested placing lower-cost QLC SSDs between TLC SSDs and hard drives and found it improved performance, prompting broader hyperscaler SSD adoption. At the same time, AI infrastructure increased HDD demand enough to create shortages. QLC SSDs then began filling both this new storage tier and, in some cases, replacing unavailable hard drives at much higher cost. That helped push NAND into shortage as well, benefiting Kioxia and SNDK.
5. Better algorithms do not necessarily mean lower hardware spending
Handy pushed back on the idea that every model-efficiency breakthrough reduces memory or infrastructure demand. When a new technique cuts memory requirements materially, hyperscalers can simply use the same infrastructure budget to process more tokens. If an algorithm improves efficiency by 6x, the result may be 6x more output rather than lower capex. Efficiency changes the productivity of AI infrastructure without necessarily reducing the dollars spent on it.
6. The HBM base die is becoming an increasingly attractive place for processing
HBM already includes a logic base die beneath the DRAM stack, making it a natural place to add functionality without creating an entirely new memory architecture. Handy expects increasingly sophisticated uses of this logic, particularly where processing closer to memory can reduce data movement. Fully custom memory remains harder economically because customers expect memory to stay cheap, while using the base die avoids the huge development cost of a fully custom DRAM design.
7. Software remains the largest barrier for alternative memory architectures
The software ecosystem for processing-in-memory and other alternative architectures remains weak. NVDA’s advantage came partly from investing early and aggressively in a complete software stack around its hardware. New memory-centric architectures either need to fit into existing programming environments or build something similarly usable. Without that, even technically strong products can struggle to reach enough volume to drive costs down. Handy cited INTC’s Optane as an example of interesting technology that never achieved sufficient scale.
8. The memory wall is increasingly about cost as well as bandwidth
The traditional memory wall is the performance penalty from constantly moving data between memory and compute. Handy believes a second wall has emerged because memory itself has become extremely expensive. Processing nearer to the data helps both problems by reducing traffic across the interface. Rather than moving an entire dataset back to the processor, smarter memory could perform an operation locally and return only the required result.
Evolving memory architectures for AI
Speaker: Ragu, Micron HBM Design Architecture
Summary: MU focused on why HBM’s bandwidth advantage increasingly comes with penalties in silicon consumption, thermals, packaging complexity and reliability. It also covered stack-height limits, workload-specific HBM and future D2D interfaces.
1. HBM’s capacity displacement versus DDR is getting worse
For the same memory capacity, an HBM die is much larger than DDR because more silicon is devoted to parallel data paths, TSVs and power delivery. MU has previously quantified HBM3E at roughly 3x the silicon of DDR5 for the same bits. Ragu said that ratio is not improving and is likely getting worse as bandwidth increases. More parallelism means larger dies and more supporting circuitry, while taller stacks create additional yield challenges.
2. Memory can represent roughly 90% of the silicon in a high-end AI package
In a representative GPU package with four 12-high HBM stacks, total memory silicon can be roughly 8x the amount of GPU silicon. That means around 90% of the silicon in the system-in-package is memory-related. The reliability implications are substantial because one failing DRAM die can affect the entire package. Ragu described HBM as one of MU’s most cross-functionally complex products, spanning process, packaging, design, test and system architecture.
3. Thermals are now a first-order architectural constraint
The HBM base die is typically one of the hottest areas because it contains the highest-speed D2D links and increasingly complex logic. Heat is generated at the bottom while cooling sits above the DRAM stack, forcing heat through every layer. Each additional DRAM die adds thermal resistance, while more base-die functionality adds heat. MU said HBM is increasingly being architected around thermal limits from the start, with localized power density and hotspots becoming as important as total power.
4. HBM has already been a meaningful source of training interruptions
Ragu cited META’s published Llama 3 work, where roughly 17% of unintended training interruptions were attributed to HBM. Reliability becomes harder as the amount of memory silicon and number of dies per package rise. HBM3/HBM3E added metadata for CRC and system-level ECC, alongside on-die symbol-based ECC and Reed-Solomon protection. MU expects RAS mechanisms to keep advancing as stacks get taller and interfaces get faster.
5. Sixteen-high looks achievable, but the path beyond that is less clear
MU sees a good technical path to 16-high HBM. Beyond that, die thinning, mechanical stress and thermal resistance become much harder. JEDEC has discussed 20-high stacks, but Ragu said substantial work remains after 16. Signaling distance itself is not the primary issue because the entire stack remains below roughly one millimeter; the tougher constraints are TSV density, power density, thermals, mechanical integrity and manufacturing yield.
6. HBM’s bandwidth scaling comes from extreme parallelism
HBM uses similar DRAM cell technology to conventional memory but accesses far more banks in parallel. HBM3E had 128 banks per DRAM die, while HBM4 moves to 256. The external interface also doubles from roughly 1,000 I/O lines to 2,000 within about the same shoreline. That has helped HBM scale from 128 GB/s and 1 GB in HBM1 to more than 2.8 TB/s and over 24 GB per cube in HBM4, at the cost of much higher silicon intensity.
7. Future HBM may become more workload specific
HBM and GPUs historically served as relatively broad, one-size-fits-all products. MU now sees growing interest in customized memory architectures for specific AI workloads. Different applications may prefer different bank structures, channel counts, access granularities or bandwidth/capacity balances. Ragu expects the processor and memory paired with it to become increasingly co-optimized as AI workloads segment.
8. High-speed interfaces and packaging are becoming part of the core memory architecture
MU expects continued innovation in memory-specific D2D interfaces, interposers and eventually optics. HBM bandwidth only matters if data can move efficiently into the processor. Packaging is also shifting from micro-bumps and thermal-compression bonding toward fusion and hybrid bonding at single-digit-micron pitches, which can increase data density while lowering thermal resistance.
Samsung on the evolution of the HBM base die
Speaker: Sanghan, Samsung Memory Business
Summary: Samsung laid out a three-stage roadmap that turns the HBM base die from a communications layer into a customized AI platform, eventually culminating in vertically stacking HBM directly on the XPU through ZHBM.
1. ZHBM could materially change the power and bandwidth equation
Samsung’s long-term ZHBM architecture eliminates the traditional 2.5D interposer and vertically integrates HBM directly on top of the XPU. Removing the lateral PHY/D2D path lets I/O and TSV structures spread across the footprint rather than clustering at the edge. Samsung is targeting roughly 0.5 pJ/bit, ~70% lower total DRAM power versus HBM5 and >2.3x higher DRAM bandwidth. Absolute DRAM power could fall by >100W, freeing that power for GPU compute. Thermal limits currently point to roughly four-high stacks rather than 12-high or 16-high.
2. Moving the memory controller into HBM could reclaim 5–10% of XPU area
Samsung sees the memory controller as one of the clearest functions to move from the XPU into the custom-HBM base die. The controller can consume roughly 5–10% of XPU silicon area, and Samsung estimates reclaiming that space for compute could potentially improve performance by 10–20%. It said most customers are actively exploring this architecture. The embedded controller can also use SRAM as a fine-grained repair resource for failing DRAM cells.
3. Custom HBM could become a gateway to a second memory tier
Samsung believes capacity is becoming nearly as important as bandwidth as context windows and KV caches expand. Its Phase 2 roadmap uses unused shoreline around the custom-HBM base die to connect directly to a second memory tier through a dedicated controller and PHY. Samsung expects better latency and bandwidth than PCIe-based expansion and said this second tier could be LPDDR or even HBF.
4. Attention is the first major compute block Samsung wants to move into HBM
Samsung does not think it makes sense to place large amounts of dense compute inside the base die because of thermal limits. Instead, it wants to selectively move memory-bound work closer to memory. For LLM inference, attention is an attractive function to offload into advanced HBM, while compute-heavy prefill and FFN remain on the XPU. The goal is to cut interposer traffic, D2D bandwidth and data-movement power without turning HBM into a full accelerator.
5. Proprietary HBM interfaces may be favored over UCIe
Replacing the traditional HBM PHY with a smaller specialized D2D interface can reduce footprint, shorten the channel and improve energy efficiency. The downside is much higher power density in a smaller area, which Samsung said was a major challenge in custom HBM4. Its Heat Path Block can reduce peak temperature by >35% when covering enough of the hot interface region. Samsung has evaluated UCIe but currently sees proprietary HBM-vendor interfaces as more attractive because UCIe is larger and consumes more power.
6. Advanced-node base-die logic is becoming mandatory
HBM energy efficiency improves every generation, but total power still rises because bandwidth scales faster. With HBM4 doubling I/O count and increasing speed, Samsung said a conventional DRAM process could leave the base die consuming more power than the DRAM core stack itself. Moving the base die to advanced logic is therefore becoming essential, while also freeing area for controllers, telemetry, test logic and other SoC-like functionality.
7. Custom HBM allows richer telemetry, testing and repair
Samsung plans to use additional base-die logic for voltage, process and aging telemetry and has integrated on-die automatic test functionality that can generate its own test patterns. A custom memory controller can therefore test DRAM more extensively from wafer level through the completed system-in-package and use SRAM resources to repair some defects discovered later in manufacturing.
8. ZHBM requires packaging and DRAM/SoC design to converge
ZHBM requires wafer-level integration and hybrid copper bonding at sub-six-micron pitches. Samsung also believes the historical separation between DRAM and SoC design has to disappear. Once the HBM base die contains custom logic, controllers and potentially compute, the memory and XPU need to be co-designed around routing, thermals, power density and physical interconnect from the start.
Advanced packaging for HBM
Speaker: Jasik Lee, SK hynix USA
Summary: SK hynix focused on the packaging needed to scale HBM toward 16-high and eventually 20-high stacks, with die warpage, inter-die spacing, hotspots and customer-package integration becoming increasingly important constraints.
1. Hybrid bonding becomes increasingly attractive around 20-high
SK hynix expects HBM eventually to move from micro-bump stacking to hybrid bonding, although Lee does not expect it for HBM4E. Removing the gaps between DRAM dies allows the silicon itself to become thicker within the same package-height limit. In a hypothetical 20-high stack, SK hynix estimates die thickness could increase roughly 20–24%, improving manufacturing margin and thermals. It also sees around a 35% improvement in thermal conductivity at 20-high.
2. Sixteen-high already requires substantially thinner dies and smaller gaps
In its 16-high HBM3E test vehicle, SK hynix still had to make individual DRAM dies roughly 10% thinner than 12-high and reduce inter-die gaps by about 50%, even after increasing total cube thickness from 720 to 775 microns. That makes die warpage and gap filling much harder. Higher pin speeds also increase power density around the PHY, adding localized hotspots to the broader stack-level thermal problem.
3. SK hynix’s molded process offers a productivity advantage
SK hynix’s process places the full DRAM stack, joins it during a single reflow and then fills the gaps with molding material. Compared with bonding each die individually, this improves productivity and allows materials with relatively good thermal conductivity between dies. The downside is greater sensitivity to warpage because individual dies can be around 50 microns thick or thinner, while ever-smaller gaps become harder to fill reliably.
4. Hybrid bonding does not solve localized base-die hotspots
Even if hybrid bonding improves overall thermal conductivity, the base die still contains small regions with extremely high power density. SK hynix is developing IHBM, which adds silicon heat blocks above hotspot regions to improve local heat extraction. Lee said this requires co-optimization from the beginning of the design and cannot simply be added late in a product cycle.
5. HBM suppliers increasingly need to co-design with customer packaging
Advanced HBM experiences much more of the customer’s subsequent packaging process than traditional memory. Hybrid-bonded HBM can be among the first components attached to an interposer, exposing it to the thermal and mechanical stress of later integration. Customers are also evaluating more interposers, bridges and alternative packaging technologies, requiring deeper collaboration around process flows, materials and mechanical stress.
6. Training and inference may ultimately use different memory architectures
SK hynix does not assume every AI workload should simply use taller HBM. Training often needs both more bandwidth and more capacity, supporting continued stack-height increases. Inference may use a smaller pool of very-high-bandwidth memory and place less bandwidth-sensitive data in LPDDR or another lower-cost tier. Lee said the industry still needs to determine the optimal architecture rather than assuming training and inference converge on the same solution.
7. HBM4 is approaching practical limits on total cube thickness
Through HBM3E, standard HBM thickness was roughly 720 microns. HBM4 moves to around 775 microns, which SK hynix described as close to the practical limit under current packaging approaches. Future scaling therefore cannot rely indefinitely on making the package taller and will increasingly depend on thinner dies, smaller gaps and eventually hybrid bonding.
8. Interposer routing could become another scaling constraint
Higher signal speeds require tighter bump pitches, more shielding and potentially more metal layers in the interposer or bridge. Those layers are much thicker than in conventional logic, so adding more can materially change wafer thickness and mechanical behavior. SK hynix said it is not yet clear how far the industry can push routing-layer counts before warpage and manufacturability become limiting.
Raptor: a 3D-DRAM accelerator for generative inference
Speakers: Ayush Ankit, META and formerly d-Matrix; Sudip Hoja, d-Matrix
Summary: d-Matrix argued that conventional HBM faces fundamental power and package-edge limits as inference bandwidth requirements rise. Its Raptor architecture stacks custom DRAM directly beneath compute and reports 100-TB/s measured bandwidth at much lower data-movement energy.
1. HBM faces both a bandwidth wall and a power wall
d-Matrix used Kimi K3 as an example of a model approaching three trillion parameters, requiring roughly three terabytes for weights, while long-context inference can add another terabyte of KV cache. HBM offers far more capacity than SRAM but becomes difficult to scale toward SRAM-like bandwidth around 100 TB/s. Pin speed, I/O width and the number of HBM stacks improve relatively slowly, while package edge limits how many interfaces can attach to compute. At current HBM energy per bit, d-Matrix estimates 100 TB/s would require roughly two kilowatts just for memory movement.
2. Raptor reports 100 TB/s at a measured 0.37 pJ/bit
Raptor uses a TSMC 4nm compute die stacked face-to-face with custom DRAM at roughly 36-micron pitch, with DRAM banks physically aligned to compute engines. d-Matrix compared this with HBM4 at roughly 2.4 pJ/bit just to move data into the base die. Its measured result is 0.37 pJ/bit, or roughly 6–7x better energy efficiency, enabling 100 TB/s in a single-high stack. Even then, full-bandwidth I/O still consumes around 300W.
3. Decode-heavy and agentic inference are the primary targets
d-Matrix separates inference into compute-heavy prefill and bandwidth-heavy decode. Prefill processes many tokens together and keeps compute highly utilized, while decode repeatedly streams weights and KV-cache data for one or a few tokens. Agentic workloads combine both phases repeatedly. Across the workloads analyzed, a meaningful portion of wall-clock time was spent in decode attention and decode MLP, making those phases particularly suited to very-high-bandwidth 3D DRAM.
4. A rack-scale deployment can hold a roughly three-trillion-parameter model
The major trade-off versus HBM is lower memory capacity per device. d-Matrix argues rack-scale aggregation can compensate. A representative configuration of around 72 cards with single-high 3D DRAM can hold a Kimi K3-class model while leaving capacity for KV cache and batching. Larger models could extend across multiple racks or use disaggregated memory.
5. Thermal behavior is manageable at one-high but difficult for multi-high stacks
Raptor puts the logic die on top so it can sit directly beneath a cold plate. The DRAM is designed to operate around 105°C and refreshes roughly 8x more frequently than standard DRAM. The current design is only one-high. d-Matrix’s simulations suggest four DRAM layers on top of a GPU dissipating more than roughly 1.3 W/mm² could push DRAM temperatures above 140°C. Logic-on-top is better for cooling; logic-on-bottom is better for power delivery.
6. Faster refresh can be handled with relatively little bandwidth loss
d-Matrix uses very small microbanks, allowing 8x faster refresh to consume only around 1.37% of memory bandwidth. Reliability is handled with Reed-Solomon coding plus CRC, while yield is improved through roughly 8–9% spare banks that can replace failing banks without sacrificing the memory channel.
7. Density is around half HBM4 today, but much of the gap may be process related
Raptor’s 3D DRAM currently delivers around half HBM4’s capacity per square millimeter. Some of the penalty is structural, but d-Matrix says a meaningful portion comes from using a less advanced DRAM process. A leading DRAM node could narrow the density gap substantially. The company reported roughly 3,000 tokens/sec/user on GLM 5.2 and 988 on the larger model using its single-high implementation.
High-bandwidth flash in AI compute
Speakers: Anurag Agarwal, Oxmiq Labs; Radha Krishna Giduturi, Praxmudi
Summary: The presentation evaluated HBF as a high-capacity AI inference tier. Simulations show it can work well in specific low-bandwidth or capacity-constrained situations, but HBM remains much more economical for high-throughput rack-scale inference.