Every Mask in a Transformer, Untangled

If you read enough about transformers, you hit the word mask everywhere — causal mask, padding mask, attention mask, loss mask, masked language modeling, masked diffusion — and it quietly starts to feel like one slippery concept you never quite pinned down.

Here’s the secret: it isn’t one concept. “Mask” names at least four completely unrelated mechanisms. A single token, in a single training step, can be masked in several different senses at once, and they have nothing to do with each other. That overloading is the entire source of the confusion.

This post untangles all of them: what each mask is, why it exists, what you get from it, and — the part most explanations skip — what breaks if you don’t use it. By the end, “mask” should never be a fog word again.

What a mask actually is

Strip away the jargon and every mask is the same primitive:

Four purposes, four families:

If you want an everyday anchor for each: SEE is a pair of blinders (limiting the field of view), SCORE is a grading rubric (which answers get marked), HIDE is a fill-in-the-blank quiz (erase something, then learn to reconstruct it), and DROP is practicing with random teammates benched (so no single neuron becomes indispensable).

Keep that table in your head and every “mask” you meet drops cleanly into one slot.

Family 1 — SEE: attention masks

What it gates: for each token (a query), which other tokens (the keys) it is allowed to attend to. Mechanically, before the attention softmax you set the score of every disallowed (query, key) pair to −∞, so its weight becomes exactly 0. That’s an N×N grid of allowed/blocked — and the shape of that grid is the whole story.

Flip through the shapes:

Causal (the decoder default)

Rule: token i sees only tokens ≤ i. Why: a decoder is trained on the whole sequence in parallel, predicting each next token. If token i could see token i+1, it would peek at the very answer it’s supposed to predict — training would be cheating, and at generation time (where the future genuinely doesn’t exist yet) the model would collapse. What you get: parallel training that is mathematically identical to left-to-right generation. Without it: the model trivially “predicts” by copying the future and is useless the moment you ask it to generate. This mask is always on in GPT/Llama/Qwen — it’s what makes them autoregressive.

Bidirectional (no mask)

Rule: everyone sees everyone. Where: encoders like BERT. Why: for understanding (classification, retrieval, tagging) you want each token informed by full left-and-right context. Trade-off: you give up open-ended generation — there’s no “next token” discipline. Put a causal mask on an encoder and you kneecap the thing it’s good at; drop the causal mask on a generator and you break it. The mask shape is the model’s personality.

Sliding window (local attention)

Rule: see only the last w tokens. Why: full attention is O(n²) in sequence length — brutal for long context. A window makes it O(n·w). What you get: long-context models (Mistral, Longformer) at tractable cost; information still travels far by hopping through stacked layers. Trade-off: any single layer’s reach is bounded by w.

Padding

Rule: pad tokens are never attended to. Why: GPUs want rectangular batches, but real sequences have different lengths, so you pad the short ones with filler. Without it: real tokens would attend to meaningless pad tokens and produce corrupted representations. It’s pure batching hygiene — but forget it and outputs silently degrade. (Related gotcha below: which side you pad on matters.)

Prefix-LM

Rule: the prefix is bidirectional; everything after it is causal. Where: T5’s decoder input, UL2’s “S-denoiser.” Why: you want the prompt understood with full context (like an encoder) while the generated part stays properly autoregressive. It’s the hybrid between the first two shapes.

Packed documents (block-diagonal)

Rule: causal and same-document only. Why: short training examples waste GPU if you pad them to full length, so you pack several into one sequence. But then token 5 of example A must not attend to example B — that’s cross-contamination, and it silently poisons training. A block-diagonal mask walls each example off. What you get: near-zero padding waste at full throughput (FlashAttention’s variable-length API / FlexAttention’s block_diag handle it efficiently). Without it: packing looks like a free speedup while quietly teaching the model that unrelated documents are context for each other.

Family 2 — SCORE: loss masks

Totally different gate. This one doesn’t touch attention at all — it decides which positions contribute to the loss.

Mechanism: alongside the tokens you build a labels array; masked positions get the sentinel −100 (PyTorch’s ignore_index), which contributes zero loss and zero direct gradient. The sequence loss is the average of cross-entropy over the scored positions only.

Why: in fine-tuning you concatenate [prompt][completion] into one sequence. The prompt is the input — handed to the model for free at inference. You only want it to learn to produce the completion. Without masking: if the prompt is 50 tokens and the completion 10, ~83% of your gradient goes into learning to regenerate the instruction text — capacity spent on something you never need, diluting the signal for the actual answer. What you get: every bit of learning signal aimed at the response.

Two boundary facts everyone gets wrong at least once:

• The first completion token _is_** scored.** The pair (full prompt → first answer token) is exactly “start the answer given the prompt” — the most important pair. Masking the prompt never means masking through the first completion token.
• The EOS token _is_** scored.** Score it or the model never learns to stop and rambles forever. Forgetting EOS is the classic loss-mask bug.

And a subtlety: masked-in-loss does not mean “no gradient at all.” The prompt tokens are still attended to, so gradient still flows into their representations via attention — the model learns to represent the prompt well, it just isn’t asked to generate it.

For the full mechanical deep-dive on this family specifically — the token-by-token labels = −100 construction, multi-turn assistant masking, and whether masking saves compute — this is the one family with its own companion piece.

Family 3 — HIDE: objective masks

Now the role of masking flips completely. In Families 1–2 the mask hides things from attention or loss. Here the mask creates the task: you deliberately hide part of the input and train the model to reconstruct it. The mask is the supervision.

Masked Language Modeling (BERT)

What: randomly replace ~15% of tokens with [MASK] and train the model (bidirectionally) to predict the originals. Why: it manufactures a supervised task from raw text while letting every token use both-side context. The 80/10/10 wrinkle: of the chosen 15%, only 80% become [MASK], 10% become a random token, 10% are left unchanged — because [MASK] never appears at inference time, so training purely on it would create a train/serve mismatch. What you get: strong bidirectional representations for understanding tasks.

Span corruption (T5)

What: mask contiguous spans (avg length ~3, ~15% of tokens), replace each with a single sentinel, and have the decoder emit the missing spans. UL2 generalizes this into a menu of denoisers (R = normal spans, S = prefix-LM, X = extreme/long spans). Why: spans force the model to generate multi-token chunks, closer to real generation than single-token MLM.

Masked diffusion LMs (the 2026 twist)

What: the forward process masks a random fraction t ~ U[0,1] of tokens; the reverse process learns to predict all masked tokens at once, and generation runs it iteratively — start fully masked, unmask in parallel over a handful of steps (the component above). Why it’s exciting: decoding many positions per step instead of strictly one-at-a-time. Gemini Diffusion (Google, 2025) reported ~1,479 tokens/second — roughly 5× a comparable autoregressive model. LLaDA showed the recipe scales to compete with strong AR baselines. Same masking primitive as BERT, aimed at generation instead of understanding.

Masked autoencoders (vision, for contrast)

What: hide ~75% of image patches and reconstruct them (MAE). It’s the same idea one modality over — proof that “hide input to create a task” is a general recipe, not a text trick.

Family 4 — DROP: regularization masks

The loosest use of the word, included so the field guide is complete. Dropout applies a random binary mask that zeroes a fraction of activations during training only (scaling the survivors to compensate). Why: it stops neurons from co-adapting into fragile combinations, which reduces overfitting. Key difference from the others: it’s random, it’s for regularization, and it’s off at inference — the other three families are deterministic and structural. (DropPath / stochastic depth is the same idea applied to whole layers.)

One token, five masks at once

Here’s the payoff — why the word felt so slippery. Take a single completion token, sitting in a packed, padded, multi-example SFT batch, and watch how many independent masks touch it simultaneously:

Five separate gates on one token, spanning three of the four families (three SEE, one SCORE, one DROP) — and if it were a diffusion-LM token instead of an SFT one, HIDE would make it all four. Nobody was ever confused about one thing called masking — they were quietly conflating a fistful of them.

Which model uses which

Gotchas & confusions

The traps, collected — most of these are silent (no error, just worse results):

• Attention mask ≠ loss mask ≠ MLM mask. The single biggest confusion. They gate different things (what you see vs what you’re scored on vs what’s hidden to learn). Always ask which family.
• Masked ≠ hidden, masked ≠ no-gradient. A loss-masked prompt token is still seen (attention) and still gets gradient through attention — it’s just not a prediction target.
• Forget to score EOS → the model never stops. Include the end token in the completion loss.
• Don’t mask through the first completion token — that transition (prompt → first answer token) is the most important thing to learn.
• Packing without a document mask → silent cross-contamination. Looks like a free speedup; actually teaches the model that unrelated examples are each other’s context.
• Padding side matters. For causal decoders you typically left-pad at generation time (so the real last token is at the true end and positions line up), while training often right-pads. Mixing them up produces garbage or misaligned outputs.
• [MASK]** train/serve mismatch** — the reason for BERT’s 80/10/10: the mask token never appears at inference, so you can’t train exclusively on it.
• Don’t compare masked vs unmasked loss values. Different denominators (different token counts) — the numbers aren’t on the same scale.
• Masking doesn’t save transformer compute. Completion-only loss skips only the tiny cross-entropy at masked positions; the full sequence still runs through every layer. Masking is about signal quality, not speed. (The real cost lever is not paying for prompt tokens you don’t train on — hence packing.)
• Dropout is training-only. Leaving it on at inference makes outputs noisy and non-deterministic.

The one-line mental model

A mask is a 0/1 gate. When you meet one, don’t ask “what’s a mask?” — ask “ignore _what**, for **what purpose_?” The answer lands it in one of four families, and everything else follows.

Get that reflex and every mask in every paper — causal, padding, block-diagonal, MLM, diffusion, dropout — is just a different answer to the same two questions.