Sizing the KV-cache block pool vs. batch size with paged attention?

@HumbleBee14·Jul 17, 2026Q&A

When serving an LLM with paged attention (vLLM-style), how should I think about sizing the KV-cache block pool vs. batch size?

I keep hitting CUDA out of memory under bursty load. A few specific questions:

  1. Is it better to cap max_num_seqs or lower gpu_memory_utilization?
  2. How do block size (e.g. 16 vs 32 tokens) and fragmentation interact?
  3. Any rule of thumb for headroom to avoid preemption/recompute?

Numbers or war stories welcome.

📌 Accepted answer
@HumbleBee14·Jul 17, 2026

Rule of thumb that has worked for me on A100/H100:

  • Lower gpu_memory_utilization first (e.g. 0.90 → 0.85) before capping max_num_seqs. Capping sequences throttles throughput harder than giving the block pool a little headroom.
  • Block size 16 is a good default. Larger blocks (32) reduce metadata overhead but waste more on short sequences (internal fragmentation). Only go bigger if your median sequence is long.
  • Keep ~10-15% KV headroom so bursty arrivals don't trigger preemption + recompute — recompute is what usually shows up as a latency cliff, not OOM.

If you are still OOMing under bursts, it is almost always the peak concurrent sequence count, not steady state. Log num_preemptions — if it is climbing, you are over-subscribed.

1 reply