Long-output reasoning has transformed the key-value (KV) cache into a critical memory bottleneck for efficient Large Language Model (LLM) serving. Conventional KV compression methods typically rely on a predefined per-request budget, keeping the total capacity fixed throughout the decoding process regardless of actual need.
Addressing Variable Demand
The research highlights that reasoning workloads exhibit substantial demand variation. Different requests necessitate different KV capacities, and the attention demand of an individual request often evolves as generation progresses. Static budgeting fails to capture these fluctuations, leading to inefficiencies in memory utilization.
The GrowPage Framework
GrowPage introduces an on-demand KV budgeting framework that treats KV capacity as a runtime resource. To manage this dynamically, the system maintains lightweight dual-timescale query summaries. These summaries capture both recent and long-term attention behaviors to estimate demand evolution through relative attention working sets.
Dynamic Allocation and Integration
At each capacity boundary, GrowPage makes a strategic decision: it either compresses KV states within the current allocation or acquires an additional physical page when broader demand is detected. By integrating with PagedAttention's page-level memory abstraction, GrowPage successfully preserves essential serving features such as continuous batching and prefix caching. Experiments across multiple reasoning benchmarks demonstrate that GrowPage achieves a superior performance-throughput trade-off compared to existing static approaches.
