ByteDance Researchers Identify Cause of Inconsistent Long-Context Retrieval in DeepSeek Models

Researchers from ByteDance's Seed team have identified a mechanism responsible for inconsistent long-context retrieval performance in DeepSeek models. Their study found that the method of compressing the KV-cache into segments, known as chunked KV-cache compression, renders retrieval accuracy sensitive to the specific position of information within a compression window. This sensitivity can result in accuracy variations of up to 40 percentage points across different positions.

The compression technique aims to reduce memory and attention costs by grouping consecutive tokens into fewer cache entries. However, the ByteDance team discovered that the same piece of information might be easily retrievable from one location while proving difficult to access from another, a phenomenon they termed "phase sensitivity."

This behavior was reproduced in models trained from scratch, where the researchers observed that different components of the attention mechanism specialized in retrieving information from particular positions. These findings help explain how strong average benchmark results can mask underlying, recurring weaknesses in the models' ability to retrieve specific details from long contexts.

The research offers crucial insights into the limitations of current large language models when processing extensive textual data and may inform future advancements in developing more reliable long-context understanding capabilities.