New Memory Solutions Boost AI Processor Throughput by 82%
Researchers have introduced V-Die and MOSAIC memory integration schemes to address heat dissipation and bandwidth bottlenecks in AI accelerators. V-Die demonstrates an 82.43% performance increase over HBM4.
A research team has unveiled two novel memory integration solutions, V-Die and MOSAIC, designed to tackle the heat dissipation and bandwidth limitations inherent in AI accelerators. These approaches aim to significantly enhance the performance of artificial intelligence hardware.
The V-Die solution's core innovation involves placing DRAM chips vertically instead of the traditional stacked configuration, thereby reducing thermal stress. It eliminates Through-Silicon Via (TSV) structures, opting for I/O connections on the bottom edge of each chip. Liquid cooling channels are integrated between adjacent chips.
According to the study, V-Die achieved a throughput of 540 tokens per second in a GPT-3 scale workload, an 82.43% improvement over HBM4's 296 tokens per second. Furthermore, memory read latency decreased by 37%, and simulations indicated a 32% reduction in first-token latency compared to H100-class hardware.
The MOSAIC solution, led by a team from the University of Tokyo, focuses on improving the manufacturability of side-oriented stacked chips. It employs orthogonal die stacking and non-contact die interconnects, utilizing microscopic induction coils instead of precise metal contacts. This solution has been reported to enable up to double the capacity of HBM4 in DRAM-on-GPU configurations.