IT Home Releases Step 5 Preview Model Weights, Supporting One Million Tokens
IT Home has released the weights for its Step 5 Preview model, which supports a one million token context window and is designed for long-range agent tasks.
Chinese technology company IT Home has officially released the weights for its Step 5 Preview large language model. The model is specifically designed for long-range agent tasks and features a context window capable of handling up to one million tokens.
The Step 5 Preview utilizes a sparse MoE (Mixture of Experts) architecture, boasting 600 billion total parameters with only 27 billion active parameters per token. This design allows for efficient processing of extensive information. The model natively supports both text and visual inputs and has demonstrated strong performance in areas such as AI programming, software engineering, and financial applications.
Model weights are available for download on Hugging Face and ModelScope, with support for various chip platforms and inference frameworks. Developers can leverage tools like vLLM or SGLang to deploy the model and integrate it into their own AI applications and workflows.
A technical blog accompanying the release details the architecture and training optimizations. Key design elements include a narrow and deep 92-layer architecture for extended computational paths, Sparse GQA technology to reduce the computational overhead of the million-token context, and a more stable training process. The model has also undergone reinforcement learning specifically for long-range tasks.