On NVIDIA's journey to firmly establish its trillion-dollar market value, two key technical supports are indispensable. One of them is the CoWoS advanced packaging led by TSMC, and the other is the HBM (High Bandwidth Memory) that is sweeping the current market.
The NVIDIA H200 is the first AI accelerator card to adopt the HBM3E memory specification. With the faster memory speed and larger capacity of HBM3e, the NVIDIA H200 provides 141GB of memory at a speed of 4.8TB per second. Compared with the A100, the capacity is almost double, and the bandwidth has increased by 43%, thereby accelerating generative AI and large language models, and improving the workload of high-performance computing (HPC).
With the rise of artificial intelligence, HBM has become a highland that giants are scrambling to seize. Storage giants such as Samsung, SK Hynix, and Micron have all regarded HBM as one of their key production products.
The popularity of HBM has caused huge waves in the market. Let's take a look at the twists and turns that the HBM boom has brought to the market.
01
The rise of HBM brings three major impacts
HBM has become one of the key words to save the industry from decline
Under the continuous downturn for several quarters, the top storage manufacturers have successively shown a trend of quarter-on-quarter growth in revenue, among which SK Hynix's performance is the most eye-catching, which is inseparable from the demand for DRAM driven by HBM.In Q1 of this year, SK Hynix's revenue reached a historical high for the same period, and its operating profit also achieved the second-highest for the same period since the best market conditions in 2018. The company sees this as a sign that it has emerged from a long period of doldrums and is now entering a comprehensive recovery phase. SK Hynix stated: "With the leadership in memory technology such as HBM aimed at AI, the company has increased product sales for AI servers, while continuing to implement profit-oriented business activities, thus achieving a sequential increase of 734% in operating profit."
Samsung Electronics has previously stated that the application of generative AI market will lead to a rapid increase in demand for memory products such as HBM that support large-scale data processing, and the company has received a large number of customer demands. It is expected that the demand for HBM may experience a steep increase in 2024.
Micron has indicated that the DRAM capacity of AI servers is 6 to 8 times that of ordinary servers, and the NAND capacity is 3 times that of ordinary servers. The DRAM capacity required by Nvidia DGX GH200 is hundreds of times that of ordinary servers.
General-purpose DRAM may face a shortage and price increase
The supply shortage of general-purpose DRAM memory chips is mainly due to the industry's large investment in DRAM types such as HBM, while the utilization rate of general-purpose DRAM production capacity is relatively low. Currently, the utilization rates of Samsung and SK Hynix are only 80% to 90%, far below the full-speed production state of NAND flash memory.
Since the beginning of 2024, the production capacity of general-purpose DRAM has only increased by about 10%, but the growth rate of the smartphone, PC, and server markets is expected to be only 2% to 3%. The reduction in investment in AI infrastructure by global cloud computing and technology companies has not significantly driven the recovery of DRAM demand.
However, during the same period, the demand for enterprise-grade solid-state drives (eSSD) has surged due to the widespread adoption of artificial intelligence. As a result, major manufacturers such as Samsung and SK Hynix have been running their NAND production lines at full capacity in the second quarter. In addition, after market conditions improved, Kioxia ended its production cut and the utilization rate of NAND production capacity returned to 100%.
Due to the significant role of general-purpose DRAM memory chips in the consumer market, such as personal computers and mobile terminals, a large-scale shortage may occur as market demand increases. According to the latest report from Morgan Stanley, the global memory market will face an unprecedented supply-demand imbalance in 2025, with a supply shortage rate of -11% for HBM and a supply shortage rate of up to -23% for the entire DRAM market.
The battle is not just about technology
The battle for dominance in the HBM technology field among memory giants is becoming increasingly fierce, from HBM1 to HBM2, HBM3, and the latest HBM3e. The evolution of each generation of technology signifies a significant improvement in storage performance and intensified market competition.HBM1 was first introduced in 2014 by AMD and SK Hynix as a competitor to GDDR, featuring a 4-stack die with a bandwidth of 128GB/s, 4GB of memory, and significantly outperforming the GDDR5 of the same period.
HBM2 was released in 2016 and officially launched in 2018, with a 4-stack DRAM die, now mostly 8-stack dies, offering a bandwidth of 256GB/s, a transfer speed of 2.4Gbps, and 8GB of memory.
HBM2E was announced in 2018 and officially proposed in 2020, with significant improvements in transfer speed and memory, providing a transfer speed of 3.6Gbps and 16GB of memory.
HBM3 was released in 2020 and officially launched in 2022, with an increase in the number of stacked layers and management channels, offering a transfer speed of 6.4Gbps, with the highest transfer speed reaching 819GB/s, and 16GB of memory.
HBM3E is an enhanced version of HBM3 released by SK Hynix, offering a transfer speed of up to 8Gbps, a capacity of 24GB, and mass production is expected to start in 2024.
However, this is not just a simple technical competition, but also an extremely fierce competition for production capacity.
According to the three major memory giants, this year's HBM supply capacity has been completely exhausted, and most of next year's production capacity has also been sold out. According to professional institutions, the dynamic gap in HBM demand for this and next year is about 5.5% and 3.5% of the production capacity. Therefore, the three major manufacturers have started a production capacity sprint competition. For example, SK Hynix is significantly expanding the production of the 5th generation 1b DRAM to cope with the increased demand for HBM and DDR5 DRAM. In terms of wafer input, the company plans to increase the monthly production capacity of 1b DRAM from 10,000 pieces in the first quarter of this year to 90,000 pieces by the end of the year, and further increase to 140,000-150,000 pieces in the first half of next year, which is 14-15 times the production capacity of the first quarter of this year. Samsung said at the end of March that it expects this year's HBM production capacity to increase to 2.9 times that of last year. Micron is building an advanced HBM testing production line in the United States and considering producing HBM in Malaysia for the first time to seize more demand brought by the AI boom.
02
Compared to GDDR, where is HBM strong?According to product classification, DRAM can be divided into DDR, LPDDR, GDDR, and HBM. The first three types are mainly used in traditional cyclical fields, while HBM is primarily driven by the AI market. Specifically, DDR is mainly used in consumer electronics, servers, and the PC sector; LPDDR is mainly used in mobile devices, smartphones, and the automotive sector; GDDR is mainly used in GPUs for image processing.
As data volume grows increasingly large and AI chips accelerate development, the von Neumann computing architecture issue becomes prominent: the performance mismatch between "storage" and "computation" causes the growth of computer computing power to encounter a bottleneck. Although multi-core parallel acceleration technology can improve computing power, the limitation of storage bandwidth still restricts the performance improvement of computing systems. GDDR is currently a widely used graphics memory technology, but in the field of AI computing, GDDR also finds it difficult to shoulder heavy responsibilities, so manufacturers turn their attention to HBM technology.
HBM has the advantage of high bandwidth. Through multi-layer stacking, HBM can achieve a higher number of I/O, allowing the graphics memory bit width to reach 1024 bits, almost 32 times that of GDDR, significantly increasing the graphics memory bandwidth. The significant increase in graphics memory bandwidth has solved the "memory wall" problem of AI computing in the past, and HBM is gradually increasing its penetration rate in high-end data center GPUs.
HBM has the advantages of high density and small size. Compared with traditional DRAM, HBM can accommodate more storage units within the same physical space, thus providing higher storage capacity. This is crucial for storing large models with billions of parameters or even larger scales. In addition, HBM achieves vertical stacking packaging of DRAM dies through 3D packaging processes, which can save a large amount of area occupied by storage chips on the chip. The size of the HBM chip is 20% smaller than that of traditional DDR4 chips and saves 94% of the surface area compared to GDDR5 chips. According to statistics from Samsung Electronics, the 3D TSV process saves 35% of the packaging size compared to the traditional POP packaging form.
Affected by its structure, the total bandwidth limit of GDDR is lower than that of HBM. Total bandwidth = I/O data rate (Gb/s) * bit width / 8. To solve the problem of low DDR bandwidth, it is essentially necessary to improve the data rate and bit width of a single I/O (the number of I/Os * the bit width of a single I/O), which can be divided into GDDR monolithic schemes and HBM stacking schemes. Monolithic GDDR adopts the method of greatly increasing the single I/O data rate to improve the total bandwidth, and the single I/O data rate of GDDR5 and GDDR6 has reached 7 Gb/s to 16 Gb/s, exceeding the 6.4 Gb/s of HBM3. HBM uses TSV technology to increase the number of I/Os and the bit width of a single I/O, thereby greatly increasing the bit width. Although it maintains a lower single I/O data rate, the total bandwidth is far superior to GDDR.
The comprehensive power consumption of HBM is also lower than that of GDDR. HBM reduces the bus frequency by increasing the number of I/O pins, thereby achieving lower power consumption. Although the large amount of cache distributed on the chip can provide sufficient computing bandwidth, due to the constraints of storage structure and process, the on-chip cache occupies most of the chip area (usually 1/3 to 2/3), limiting the improvement of computing power.
Today, HBM has become an indispensable key component in core facilities such as supercomputers and data centers, providing a solid memory foundation for large-scale parallel computing. Especially in the field of graphics processing, the high bandwidth characteristics of HBM enable GPUs to access and process image data more quickly, thereby bringing users a smoother and more realistic visual experience.
Even the industry-leading Nvidia also shows a high degree of dependence on this product. It is reported that several flagship products released by Nvidia in recent years (such as A100, H100, H200) are all equipped with different amounts of HBM. HBM has become an essential partner for Nvidia's AI chips. However, about 90% of the global HBM market is monopolized by two Korean companies, SK Hynix and Samsung.Will HBM be in persistent shortage or oversupply by the end of 2024?
In the competition among the three giants of memory storage, due to the leading performance of Hynix's HBM3 products, it was the first to secure orders from Nvidia, becoming the main supplier of its server GPU.
Samsung focuses on orders from some cloud customers, while Micron directly skipped HBM3 and put its main efforts on the HBM3E product. Judging from the current situation, Micron's market share has some gap with the first two players.
But no matter what, the three memory chip giants are sparing no effort in expanding production capacity. For example, Hynix has made ambitious statements, planning to invest up to $74.8 billion by 2028, of which 80% will be used for the research and development and production of HBM, and the mass production time of the next-generation HBM4 chip will be advanced to 2025.
According to the analysis of professional institutions, the dynamic gap of HBM demand in the next two years is about 5.5% and 3.5% of production capacity.
However, according to the data from Dolphin Research, HBM is expected to shift from "short supply" at the end of 2023 to "oversupply" at the end of 2024.
Dolphin Research compared the supply and demand situation of HBM from 2023 to 2024. From the demand side, combined with the capital expenditure of cloud service providers and AI shipment, the market demand for HBM is expected to increase from 284MGB in 2023 to 512MGB; from the supply side, combined with the production capacity plan of mainstream storage manufacturers, the supply of HBM is more likely to quickly increase from 147MGB in 2023 to more than 1000MGB.
In this supply and demand relationship, the supply side in 2023 can only meet half of the demand, and there is a serious shortage of supply. The situation of supply shortage has driven various manufacturers to significantly increase their production capacity plans. If the production capacity plan is implemented as planned, the supply and demand relationship of HBM in 2024 may undergo a significant reversal, and the supply side will exceed the overall market demand.
As for the price of HBM, due to the relatively tight supply, the overall price will rise in 2024. However, as Samsung's products pass certification and the release of production capacity, the product price of HBM may also fall.Domestic Manufacturers Accelerate Breakthroughs
With the rapid development of AIGC technology applications, the demand for AI servers and high-end GPUs continues to grow, which will further promote the high-speed growth of the HBM market. It is expected that by 2025, the demand for Chinese HBM is likely to exceed 1 million units.
China, however, is still a latecomer in this race.
Not long ago, the third phase of the National Integrated Circuit Industry Investment Fund was launched. The registered capital of the Big Fund Phase III is as high as 344.4 billion yuan, which far exceeds the scale of the first two phases of the fund, showing the national attention and strong support for the integrated circuit industry. Such a huge capital injection will undoubtedly provide a strong impetus for the development of the integrated circuit industry.
At present, the focus of the external investment of the Big Fund Phase III has not been officially disclosed. It is reported that the Big Fund Phase III is expected to increase coverage of the artificial intelligence chip segment, supporting large wafer factories in the HBM industry chain.
In March of this year, Wuhan Xinxin released the bidding project of "High Bandwidth Storage Chip Advanced Packaging Technology Research and Development and Production Line Construction", which marks the company's official entry into the HBM market. By utilizing three-dimensional integrated multi-wafer stacking technology, Wuhan Xinxin aims to create storage solutions with higher capacity, larger bandwidth, and lower power consumption to meet the market demand for high-performance storage chips. In the face of overseas factories' mass production of HBM3E, domestic storage manufacturers are also accelerating breakthroughs in HBM technology, which is expected to enhance their competitive strength under the demand of the AI wave.
Facing the mass production of HBM3E by overseas giants, domestic storage manufacturers are also accelerating breakthroughs in HBM technology, which is expected to enhance their competitive strength under the demand of the AI wave.
The packaging test leader, Changdian Technology, said in investor interactions that its XDFOI high-density fan-out packaging solution is also suitable for HBM's Chip to Wafer and Chip to Chip TSV stacking applications; Tongfu Microelectronics previously stated that after the completion of the advanced packaging production line at the Nantong Tongfu factory, the company will become the most advanced 2.5D/3D advanced packaging research and development and mass production base in China, achieving a breakthrough in the field of high-performance packaging technology for HBM (high bandwidth memory) in China, which is of great significance to the country in the field of integrated circuit packaging and testing.
China's main storage chip companies are also cooperating with packaging and testing manufacturers such as Tongfu Microelectronics to carry out HBM-related projects.However, overall, domestic manufacturers are still in the early stages of development in HBM technology. Although there are more advanced HBM3 and HBM3E products internationally, domestic memory manufacturers are currently in the research and development and industrialization stage of HBM2.
Facing the mass production advantages of overseas giants in advanced technologies such as HBM3E, domestic manufacturers still need to accelerate their pace of catching up, overcome technical barriers, and achieve a transformation from technology following to technology leading. In this process, customers' continuous high demands for AI server performance, memory bandwidth, and memory size are both a huge challenge to HBM technology and a driving force for its continuous advancement.
It is worth noting that although HBM has an advantageous position in bandwidth performance, its high cost and power consumption also promote the industry to explore more diversified solutions. The rapid development of memory technologies such as GDDR and LPDDR provides more choices for AI processors, especially in application scenarios seeking the best balance between cost, performance, and power consumption. For example, the introduction of GDDR7 not only significantly improves memory capacity and data transfer rate but also becomes an ideal choice for some AI applications due to its relatively lower complexity.
In summary, as AI technology continues to evolve and market demand becomes more diversified, HBM, as a representative of high-performance storage technology, will continue to play a key role in specific fields. At the same time, domestic manufacturers need to seize opportunities, accelerate technological innovation and industrial upgrading, and meet the diversified needs of high-performance computing in the AI era with more flexible and diverse memory solutions.
Comment