Memory, not maths, sets the pace of AI, so the biggest buyer now designs it
Nvidia’s newest AI chip, Rubin, can do 35 thousand trillion sums a second. Yet when it writes an answer, one word at a time, much of that power sits idle.
Nvidia’s own engineers call that step “a memory-bound operation”. The chip finishes its sums quickly, then waits for the model’s numbers to arrive from memory.
So memory, not maths, sets how fast an AI answers and how many people one chip can serve. That has made memory the part chip makers compete on, and it is changing what memory is. For decades a memory chip was a standard part, the same for every buyer. From the next generation, the biggest buyer designs the bottom of the memory stack itself.
The memory in question is HBM, high-bandwidth memory: towers of memory chips stacked beside the processor. To write each word, the chip must read the whole model out of those towers again. A faster chip that cannot be fed faster simply waits longer.
The gap shows in the makers’ own spec sheets. Since 2020, the peak maths in Nvidia’s top chip has grown about 112 times. The speed of the memory feeding it has grown about nine times.
Explain it like I don't work in finance
A chef gets faster at chopping every year, but the ingredients still come through one hatch from the pantry. Past a point, a faster chef just stands at the hatch, and the kitchen serves only as many plates as the hatch allows.
So kitchens widen the hatch and build bigger pantries. The busiest kitchen has now gone further: it designs its own hatch, made to fit its kitchen and no other.
The chip makers’ answer has been to pile on memory. Nvidia’s H100 of 2022 carried 80 gigabytes; Rubin carries 288, and AMD plans 432 for its next chip. This year’s memory generation, HBM4, doubled the number of connections between each tower and the chip.
Doubling them needed a smarter chip at the foot of each tower, called the base die, and it is increasingly made the way processors are made. Samsung’s chip foundry credits its improving results to selling these base dies. In August Nvidia went further. It announced a memory design that moves its own controller, the circuit that runs the memory, into that base die. Micron says it is working with Nvidia on the first one.
That matters beyond engineering. A custom part is designed to one buyer’s plan, generation after generation. If demand for that design falls, it cannot simply be sold as standard memory. Memory has long been a commodity that swings from glut to shortage. Memory built to one customer’s design behaves more like a chip made to order.
My test is whether that becomes real at scale soon. By the end of 2027, at least two of the three makers, SK Hynix, Samsung and Micron, should say they are shipping custom memory in volume.
For investors
The chip waits on memory, not maths
An AI model answers in two steps. It reads the question in one pass, which keeps the maths busy. Then it writes the answer one word at a time, and for each word it must read every weight of the model from memory again.
Nvidia’s engineers describe the second step plainly. It is “a memory-bound operation”, and model execution is “frequently memory-bandwidth bound”, “in particular, bandwidth-bound in the weights”. Even after every software trick, it is “still very likely to be memory bound”.
A rough sum shows why. A model of 70 billion weights stored at eight bits fills about 70 gigabytes. At 8 terabytes a second, one full read takes about nine milliseconds, so a single conversation tops out near 110 words a second, however fast the maths. That is my arithmetic, and it ignores overlap and caching.
Serving many users at once shares each read, but every conversation keeps its own working memory, the KV cache. Nvidia’s own example puts it at about 2 gigabytes for one 4,096-token conversation on a small 7 billion weight model, and it grows with every user and every word. So bandwidth sets the speed of each answer, and capacity sets how many conversations fit on one chip.
The running measure in this piece is maths per byte: how many operations a chip can do for each byte its memory delivers. The higher it climbs, the longer the chip waits.
For six years the maths outran the memory
On Nvidia’s own spec sheets, maths per byte rose from about 150 on the A100 of 2020 to about 1,900 on Blackwell Ultra in 2025, more than twelvefold.
Since 2020 the maths has grown about twelve times faster than the memory feeding it
Peak maths and memory bandwidth per Nvidia chip, as multiples of the A100 of 2020
The caveat is real but does not reverse the direction. Held at 16-bit, peak maths rose about seven to eight times from the A100 to Blackwell Ultra while bandwidth rose about four. Smaller number formats have been the industry’s main release valve, and with the maths already at four bits, few halvings are left.
The memory makers say the same in their own words. SK Hynix’s head of memory systems research told the Open Compute summit in October that processor-centric designs “face a bottleneck known as the ‘memory wall’”.
Rubin held the line, on HBM4
Rubin is the first Nvidia generation in this run where maths per byte did not rise: about 1,820, against 1,875 for Blackwell Ultra. Its bandwidth jumped 2.4 times, to 19.2 terabytes a second, on HBM4.
That came from the standard. JEDEC’s HBM4 standard of April 2025 doubles each stack’s interface to 2,048 bits and its channels to 32, for up to 2 terabytes a second per stack. SK Hynix says its HBM4 runs above 10 gigabits a second per pin, against the standard’s top speed of 8. So a maker already runs memory past its own rulebook.
Capacity has climbed alongside. A bigger model, or more conversations, needs more gigabytes on the chip.
Memory per chip has grown 3.6 times since the H100
High-bandwidth memory on each AI chip, in gigabytes, by maker and year of launch
AMD sells on it directly: its own slide says “Highest memory bandwidth & capacity drives performance”. Micron told investors in September that every AI platform’s “value proposition is enhanced by the performance and capacity of memory”. For the buyer, memory per chip has become a headline specification, beside the maths.
The bottom of the stack became a logic chip
Wider memory needs a cleverer base die, the chip at the foot of each stack that talks to the processor. From HBM4 it moves towards logic processes, the kind used for processors in chip foundries.
The three makers have taken three routes. SK Hynix said in April 2024 that, for HBM4, it planned to “adopt TSMC’s advanced logic process for HBM4’s base die”, so it could “produce customized HBM”. Samsung makes its base die in its own foundry: its foundry business credited improving earnings to “HBM base-die demand” in July and to “HBM base-die sales growth” in October, on a 4 nanometre process. Micron, as the chip supply chain piece notes, makes its HBM4 base die in-house.
So at Samsung, and at SK Hynix if its 2024 plan held, a stack of memory now passes through a logic factory before it is finished. Samsung’s foundry results already move with HBM.
The buyer now designs the memory
The strongest evidence is a buyer’s own design. On 26 August 2026 Nvidia announced NVHBM. Standard HBM, it said, places “the memory controller on the XPU die”, using processor space that could do maths. NVHBM instead “integrates NVIDIA’s custom memory controller into the HBM base die”.
With NVHBM, Nvidia's design moves into the memory stack
Where the memory controller sits, standard HBM against Nvidia's custom design; the shaded part is what moves
Nvidia claims up to 30% more bandwidth and 15% lower memory power than standard HBM4E, and says NVHBM will be “available from multiple memory providers”. Micron named itself a month later: it is working with Nvidia on “the industry’s first custom-HBM4E implementation, NVHBM, to be adopted on next generation of GPUs”. Its annual report adds that HBM4E “customization options” bring “even deeper R&D engagement with customers”, and that HBM4E ramps in volume in calendar 2027. Samsung shipped its first HBM4E samples in the second quarter of 2026.
A custom base die changes the trade. The stack is designed to one buyer’s plan for each generation. Nvidia offers that plan to the chipmakers that link to its systems, but it stays Nvidia’s design, and if demand for it falls, the stacks cannot simply be sold as standard HBM. That is how a foundry works, and it is not how memory has worked.
Two of the three makers are Korean
Nvidia names its memory suppliers in its annual report: SK Hynix, Micron and Samsung. Two are Korean, and SK Hynix’s new HBM capacity before 2028 is in Korea.
SK Hynix will run its M15X fab in Cheongju at full capacity by the end of 2026 to make more HBM4. A dedicated HBM packaging plant in the same city, P&T7, gets its test line by October 2027 and packaging by February 2028. Its American HBM packaging plant, in Indiana, starts mass production in the second half of 2028. Micron’s HBM packaging plant in Singapore starts output in early 2027.
SK Hynix expects demand to outrun supply “through 2030”, citing “the long lead time required to build new production facilities”. A custom stack adds a logic foundry to that chain, so every stack now depends on a memory fab, a logic fab and a packaging line being ready together.
What could loosen the memory limit
The case against is real. First, smaller numbers: each halving of the number format halves the bytes per weight, and software keeps finding savings. Second, overflow: Micron says SSDs are now used “for KV cache offload”, and SK Hynix is showing high bandwidth flash as a cheaper tier. Both ease capacity more than bandwidth, since the weights still have to be read for every word.
The custom half is weaker still. NVHBM has one named chip partner, Amazon’s Annapurna Labs, “starting with Trainium4”, and no date. The HBM4 standard stays backward compatible with HBM3 controllers, and Nvidia measures NVHBM against “standard HBM4E”, which remains on sale. Custom memory could stay one buyer’s niche through 2027. That is exactly what the test below checks.
None of this lets the maths stop waiting. The chef is still faster than the hatch. What has changed is who designs the hatch: the biggest kitchen now draws its own, and the memory makers build it to order.
The stack is tied to the buyer, but the buyer is not tied to the maker
A custom part usually ties a buyer to the one supplier able to make it. NVHBM runs the other way. The design is Nvidia’s, and Nvidia says it will be “available from multiple memory providers”, so it can move that design between makers.
The maker gets no such freedom. A stack built around Nvidia’s controller fits Nvidia’s chips and the partners that link to its systems, with Amazon’s Trainium4 the first named. If demand for that design falls, the stack, as I read it, cannot be sold to anyone else as standard HBM.
With NVHBM, a stack fits one design, and the design can go to any maker
Who designs the base die under each stack, and who can use a finished one
Custom HBM itself is not new: SK Hynix spoke of “customized HBM” in 2024. Who draws the base die is the newer part. When the maker drew it, custom was the maker’s edge. When the buyer draws it, the edge stays with the buyer, and the makers compete to build its design.
Memory plants take years; the chips they feed change every year
That matters more because the makers commit to buildings years ahead. SK Hynix’s Cheongju packaging plant packages by February 2028 and its Indiana plant in the second half of 2028. Nvidia has launched a new flagship every year since 2024.
The new memory plants come on stream one to two chip launches after Rubin
Nvidia's flagship and the HBM generation by year, against the memory makers' new plants by stated date
So a line planned today will run stacks for chips a generation or two after Rubin. My reading: under standard HBM, the maker planned that line against a published standard. From HBM4E, part of what it builds is set by a buyer, one generation at a time, after the plant is already standing.
For now the shortage hides that risk. Micron has agreed most of its 2027 HBM already, as the chip supply chain shows, and SK Hynix expects demand to outrun supply through 2030. The risk shows only when that ends. Then the biggest buyer holds a part shaped to its chip and a choice of makers, while chip designers without the volume to commission a stack stay on standard parts.
Three dates will show which way it goes: Micron’s quarterly remarks as HBM4E ramps through 2027, Nvidia’s GTC in March 2027, on the claim’s watch list, for an NVHBM date or second partner, and Samsung’s and SK Hynix’s own results.
What would prove me wrong
The test fails if, by the end of 2027, fewer than two of the three makers say in their own words that they ship custom HBM in volume. That is a genuine risk: Micron’s HBM4E ramps only in 2027, NVHBM has no published date, and Samsung and SK Hynix have said nothing dated about custom volume in releases I could open.
A pass would show that buyers treat memory as a part worth designing, not prove the memory limit itself. That wider argument weakens if the chips after Rubin and MI400 stop raising memory per chip.
Research notes
Micron has an initiation note, resting on its own claim about HBM wafers. How far ahead each year’s HBM is sold is in the chip supply chain. Whether any of this is already in share prices is a question for these notes, not for the journal.
The claims on record, word for word
The test this piece stands behind, quoted word for word from the claims page:
From HBM4E, the memory beside AI chips is built to the buyer’s design (31 October 2026)
Claim: “From HBM4E, high-bandwidth memory becomes a part built to the buyer’s design: by 31 December 2027, at least two of SK Hynix, Samsung and Micron will have said, in their own results releases, filings or newsroom, that they are in volume production of, or shipping in volume, HBM whose base die carries a customer’s own design, such as Nvidia’s NVHBM.”
Breaks if: “By 31 December 2027, fewer than two of SK Hynix, Samsung and Micron have said, in their own results releases, filings or newsroom, that they are in volume production of, or shipping in volume, HBM whose base die carries a customer’s own design, such as Nvidia’s NVHBM; a maker’s own base die sold as ‘custom HBM’ does not count unless the maker says it carries the customer’s design. Samples, development, qualification, design wins and plans do not count, and nor do press or research-firm reports.”
There are no earlier claims on this subject; this is the first.
The figures behind the charts
| Chip | Launch | Memory | Bandwidth | Peak dense maths, smallest format | Maths per byte | Source |
|---|---|---|---|---|---|---|
| Nvidia A100 80GB | 2020 | 80 GB HBM2e | 2.039 TB/s | 312 TFLOPS, 16-bit (624 with sparsity) | about 153 | Nvidia A100 page |
| Nvidia H100 | 2022 | 80 GB | 3.35 TB/s | about 2 PFLOPS, 8-bit (3,958 TFLOPS with sparsity) | about 600 | Nvidia H100 page; Blackwell Ultra blog |
| Nvidia H200 | 2024 | 141 GB HBM3e | 4.8 TB/s | as H100 | about 410 | Nvidia H200 page |
| Nvidia Blackwell | 2024 | 192 GB HBM3E | 8 TB/s | 10 PFLOPS, 4-bit | 1,250 | Blackwell Ultra blog, 22 August 2025 |
| Nvidia Blackwell Ultra | 2025 | 288 GB HBM3E | 8 TB/s | 15 PFLOPS, 4-bit | 1,875 | same |
| Nvidia Rubin | 2026 | 288 GB HBM4 | 19.2 TB/s | 35 PFLOPS, 4-bit, training (50 inference, with sparsity) | about 1,820 | Vera Rubin NVL72 page |
| AMD MI300X | 2023 | 192 GB HBM3 | 5.3 TB/s | AMD MI300X page | ||
| AMD MI355X | 2025 | 288 GB HBM3E | 8 TB/s | 10 PFLOPS, MXFP4 | AMD deck, 12 June 2025 | |
| AMD MI400 | 2026, projected | 432 GB HBM4 | 19.6 TB/s | 40 PFLOPS, FP4 | same, engineering projection |
Maths per byte is peak dense operations divided by bytes a second, my arithmetic. Nvidia publishes sparse figures by default and states that dense is half. The A100 is shown at 16-bit because 8-bit integer maths was not its usual format for models; at 8-bit its maths per byte would be about 306. Rubin at 16-bit (4 PFLOPS dense) gives about 210 against about 280 to 310 for Blackwell Ultra (Nvidia’s HGX and rack pages imply 2.25 to 2.5 PFLOPS dense), so at fixed format Rubin narrowed the gap, which fits the story of HBM4 catching up a step. The V100 of 2017 is left off because Nvidia no longer publishes its specification page.
Not used: market shares and any report that SK Hynix’s shipping HBM4 base die is made by TSMC, which rest on press and research firms; SK Hynix’s own words are its plan of April 2024. Micron’s HBM4E base die partner is likewise press only.
Who is exposed if memory becomes a part built to the buyer's design
This maps who operates in each layer. It is not a recommendation, and naming a company is not a view on its shares. Figures in the piece are disclosures used as evidence for the mechanism.
- The memory makers
- SK Hynix, Samsung and Micron make HBM, the stacked memory beside AI chips.
- The base die
- TSMC and Samsung's foundry make logic base dies for HBM4.
- The buyers who design it
- Nvidia and AMD design AI chips; Amazon's Annapurna Labs is the first named NVHBM partner.
- On the other side
- Chip designers without the volume to commission their own stack, and memory makers whose standard HBM loses ground to custom parts they do not make.
- Where I stop
- I have no view on the memory equities.
Sources: Nvidia spec pages for the A100, H100, H200, DGX B200, HGX and Vera Rubin NVL72 (read 30 October 2026), its technical blogs of 17 November 2023 and 22 August 2025, its NVHBM blog (26 August 2026) and Form 10-K (25 February 2026); AMD's MI300X page, its release of 2 June 2024 and Advancing AI deck (12 June 2025); JEDEC's HBM4 release (16 April 2025); SK Hynix releases of 3, 19 and 24 April 2024, 12 September 2025, 22 April, 16 October and 28 October 2026; Samsung results of 30 April, 30 July and 29 October 2026; Micron's prepared remarks (30 September 2026) and Form 10-K (9 October 2026). Maths per byte and the read-time sum are my arithmetic. Personal research, not investment advice.