DeepSeek releases open-source inference framework for Nvidia and Huawei chips
DeepSeek has open-sourced its DeepInfer framework, allowing developers to run large-scale model services on Nvidia and Huawei hardware.
DeepSeek released its inference framework, DeepInfer, on October 20 to support the deployment of large-scale language models. The software provides technical capabilities such as distributed inference, speculative decoding, and structured output. The framework is compatible with both Nvidia GPUs and Huawei Ascend processors.
Alongside the framework, the company open-sourced deepseek-recipe, a tool designed to process API requests and handle input-output streams consistent with DeepSeek's internal service architecture. The release aims to assist businesses in improving model throughput and latency while reducing computational costs and energy expenditure.
Currently, DeepInfer supports the DeepSeek-V4.1-Flash-1017 model. The company stated it intends to continue releasing inference code and performance optimizations for future models through this project.