Sizing the KV-cache block pool vs. batch size with paged attention?
When serving an LLM with paged attention (vLLM-style), how should I think about sizing the KV-cache block pool vs. batch size?
I keep hitting CUDA out of memory under bursty load. A few specific questions:
- Is it better to cap
max_num_seqsor lowergpu_memory_utilization? - How do block size (e.g. 16 vs 32 tokens) and fragmentation interact?
- Any rule of thumb for headroom to avoid preemption/recompute?
Numbers or war stories welcome.
Rule of thumb that has worked for me on A100/H100:
gpu_memory_utilizationfirst (e.g. 0.90 → 0.85) before cappingmax_num_seqs. Capping sequences throttles throughput harder than giving the block pool a little headroom.If you are still OOMing under bursts, it is almost always the peak concurrent sequence count, not steady state. Log
num_preemptions— if it is climbing, you are over-subscribed.1 reply