Tencent Details Scaling of Large Language Model to 617 Billion Parameters
Tencent's WeChat AI team has outlined a new scaling approach for its WeLM model family, enabling models to reach 617 billion parameters.
Tencent's WeChat AI team has detailed a new scaling method for its WeLM model family. The team trained WeLM-HD4-80B and WeLM-HD4-617B models using a technique called Hidden Decoding. This method expands each token into multiple internal computation streams without altering the core Transformer architecture.
The 80 billion parameter model activates 3 billion parameters during operation, while the 617 billion parameter model activates 23 billion. In internal tests, both models outperformed their corresponding autoregressive baseline models across nine shared benchmarks.
Training costs were reported at 5.1 times the baseline for the 80B model and 4.4 times the baseline for the 617B model. This approach offers a way to enhance model performance and efficiency at scale.
The development represents a significant advancement in managing and training extremely large language models, potentially paving the way for more sophisticated AI capabilities.