The arithmetic of attention: why FlashAttention still matters
Memory bandwidth, not FLOPs, is what bounds modern inference. A walk through the numbers behind a kernel that quietly reshaped the field.
Machine learning,
from to clusters.
An open community for learning, writing, and tinkering on the infrastructure behind modern AI — inference engines, training systems, ml stacks, and everything in between.
13
articles
9
contributors
10
topics
FIG. 1.1Per-head attention pattern at layer 14. Causal mask + induction circuit + sink token, visualized over one sentence.
Latest
Recently published.
The arithmetic of attention: why FlashAttention still matters
Memory bandwidth, not FLOPs, is what bounds modern inference. A walk through the numbers behind a kernel that quietly reshaped the field.
Liam Chen Nov 12, 2026 · 14 min
Continuous batching, revisited
Three years after the original paper, what does state-of-the-art serving actually look like? A field report from a team running 12B tokens a day.
Priya Raghavan Nov 06, 2026 · 19 min
What we've been getting wrong about MoE routing
Top-k routing has become a default. It shouldn't be. A look at the tradeoffs nobody's measuring and the experiments that change my mind.
Hugo Belmar Oct 28, 2026 · 22 min
Index
Browse by topic.
-
-
Continuous batching, revisited
Three years after the original paper, what does state-of-the-art serving actually look like? A field report from a team running 12B tokens a day.
-
Speculative decoding without the speculation
Draft models work. They also fail in ways the original papers didn't surface. A small bag of tricks for keeping acceptance rates high in real workloads.
-
Notes on KV cache paging at scale
PagedAttention is a good idea poorly understood. A primer, plus the second-order effects you only see at 10,000 concurrent requests.
-
Embeddings as compression: the bitter lesson, retold
We've stopped treating embeddings like first-class data. A case for revisiting them, with measurements from a 200M-document corpus.
-
RAG is not retrieval-augmented generation
It is a feedback loop, an index policy, and a re-ranker pretending to be a system. Why most RAG postmortems mistake the symptom for the disease.
Discourse
The conversation.
GitHub Discussions
Long-form threads tied to GitHub identities. Best for technical questions, paper discussions, and feature proposals. Searchable, attributed, permanent.
Open Discussions →Discord
Real-time chat for the working day. Quick questions, debugging help, paper club, and the occasional argument about whether MoE is overrated.
Join the server →Tools
Run the math yourself.
Attention Visualizer
Inspect attention patterns layer-by-layer for any Hugging Face model. Click any head to see its causal mask, induction behavior, and sink tokens.
· Live ·
Throughput Calculator
Estimate tokens/sec for any model, precision, batch size, and GPU combination — memory-bound roofline only, no kernel-quality wishful thinking.
· Live ·
Training Memory Calculator
Per-GPU VRAM breakdown for training and inference — params, gradients, optimizer state, activations, KV cache, with ZeRO/FSDP sharding.
· Live ·
Eval Harness Playground
Run a small set of evaluations against any inference endpoint and get back a structured scorecard — quality, latency, cost, and refusal rate side by side.
· Beta ·
Model Card Generator
Generate a structured model card from a checkpoint and an evaluation log — covers intended use, training data summary, evals, limitations, and ethics.
· Beta ·
Kernel Benchmark
Side-by-side timings for Triton, CUDA, and PyTorch implementations of the same kernel — attention, layernorm, GEMM, custom — across shapes and dtypes.
· Soon ·