DeepSeek shipped two very different answers to “attention is too expensive at long context” within one model generation.
MLA (Multi-head Latent Attention) caches one compressed latent per token and reconstructs per-head keys and values from it. It carried V2, V3 and V3.2 — all three ship kv_lora_rank = 512 — and spread well past DeepSeek: GLM-4-MoE, LongCat-Flash, MiniCPM3, Mistral-4 and Kimi K3 all carry the field today. (Absorbed vs. naive MLA covers it; this post assumes it.)
Then DeepSeek moved on. V4 has no kv_lora_rank at all — just as MLA became an industry default, the lab that invented it dropped it.
V3.2-Exp is conservative: V3’s MLA untouched, plus a cheap lightning indexer that picks the top-2048 tokens each query may look at and folds that choice into an additive mask. The cache is still every token; only the read is sparse.
V4 is not. Every layer is shared-KV MQA over a 128-token sliding window plus a compressed long-range branch, the compression rate alternating layer by layer between $m=4$ (CSA, Compressed Sparse Attention, indexer-selected) and $m'=128$ (HCA, Heavily Compressed Attention, dense over blocks). The cache is a short window plus a running list of pooled entries.
Part 1 — V3.2: MLA + Lightning Indexer
Figure 2 of the V3.2 technical report. Cover the green path and what is left is absorbed MLA — hence the grey box saying Multi-Query: one cached $[\mathbf{c}^{KV}_t; \mathbf{k}^R_t]$ entry per token, not reconstructed per-head keys. The green path is all DSA adds: indexer queries branch off the same query latent $\mathbf{c}^Q_t$, the single key $\mathbf{k}^I_t$ and per-head weights come off $\mathbf{h}_t$, and the top-k selector gates which cached entries reach core attention.
The indexer
V3.2 inherits MLA from V3 verbatim. Only three config fields are new:
class DeepseekV32Config(Glm4MoeLiteConfig, RotaryEmbeddingConfigMixin):
hidden_size: int = 7168
num_hidden_layers: int = 61
num_attention_heads: int = 128
kv_lora_rank: int = 512 # MLA latent
q_lora_rank: int = 1536
qk_nope_head_dim: int = 128
qk_rope_head_dim: int = 64
v_head_dim: int = 128
# --- DSA ---
index_topk: int = 2048 # tokens kept per query
index_head_dim: int = 128
index_n_heads: int = 64
The asymmetry is what makes DSA cheap: 64 indexer heads of dim 128, but one shared key per token, so the indexer caches 128 values/token against MLA’s $512 + 64 = 576$ — under a quarter, and never up-projected.
One score per (query $t$, key $s$) pair, $j$ indexing the 64 indexer heads:
$$ I_{t,s} \;=\; \sum_{j=1}^{H^I} w^I_{t,j}\;\operatorname{ReLU}\!\left(\mathbf{q}^I_{t,j}\cdot\mathbf{k}^I_s\right) \qquad \mathcal{S}_t \;=\; \operatorname*{Top\text{-}k}_{s\,\le\,t}\; I_{t,s} $$class DeepseekV32Indexer(nn.Module):
def __init__(self, config, layer_idx):
self.wq_b = nn.Linear(config.q_lora_rank, self.n_heads * self.head_dim, bias=False)
self.wk = nn.Linear(self.hidden_size, self.head_dim, bias=False) # single shared key
self.k_norm = nn.LayerNorm(self.head_dim, eps=1e-6)
self.weights_proj = nn.Linear(self.hidden_size, self.n_heads, bias=False)
self.softmax_scale = self.head_dim**-0.5
@torch.no_grad()
def forward(self, hidden_states, q_resid, position_embeddings, attention_mask, position_ids, past_key_values=None):
cos, sin = position_embeddings
q = self.wq_b(q_resid).view(batch_size, seq_len, self.n_heads, self.head_dim) # [B, S, 64, 128]
k = self.k_norm(self.wk(hidden_states)) # [B, S, 128] one shared key
# RoPE on the leading 64 dims of both — half-split here, interleaved on the MLA path
q, k = apply_partial_rope(q, k, cos, sin)
scores = torch.matmul(q.float(), k.transpose(-1, -2).float().unsqueeze(1)) * self.softmax_scale
scores = F.relu(scores)
weights = self.weights_proj(hidden_states).float() * (self.n_heads**-0.5)
index_scores = torch.matmul(weights.unsqueeze(-2), scores).squeeze(-2) # [B, S, T]
index_scores = index_scores + attention_mask # causality first
topk = min(self.index_topk, index_scores.shape[-1])
return index_scores.topk(topk, dim=-1).indices.to(torch.int32) # [B, S, topk]
Three things worth stopping on:
- ReLU, and no softmax anywhere. $I_{t,s}$ is never normalized over $s$ — only its ranking is consumed. ReLU makes it a sum of non-negative per-head votes, so a token needs just one head to care about it to survive.
- Selection is head-uniform. The sum over $j$ collapses the head axis before the top-$k$, so all 128 attention heads read the same 2048 tokens. Absorbed MLA holds one shared latent per token, so each selected latent is fetched once and reused by every head — per-head selection would multiply that traffic by up to $128\times$, and DSA would relocate cost rather than remove it.
- The scoring path is fp32.
_keep_in_fp32_modules = ["indexer.weights_proj"], plus explicit.float()onq,kandweights. Top-$k$ is discrete: in bf16 two close scores can round together and the pick becomes arbitrary.
How the indexer is trained
Training runs in two stages: stage one freezes everything except the indexer; stage two unfreezes the model and switches top-$k$ on. In both, the indexer input is detached from the graph, so its KL objective never reaches the main model’s parameters.
What DSA actually buys
Core attention per query goes from $O(T)$ — score against every cached token — to $O(k)$ with $k = 2048$ fixed, so the cost stops growing with context. Below ~2k tokens that is pure overhead; at 128k it is a $\sim 60\times$ cut. The indexer is $O(T)$ too, but with a tiny constant: no value, no output projection.
The cache is still $O(T)$ though: 576 latent values per token plus 128 for the indexer. V3.2 makes attention sparse but not memory. That is what V4 attacks.
Part 2 — V4: Sliding Window + Compressed Long Range
Shapes for V4-Flash:
hidden_size: int = 4096 # V4-Flash
num_hidden_layers: int = 43
num_attention_heads: int = 64
num_key_value_heads = 1 # shared-KV MQA
head_dim: int = 512
q_lora_rank: int = 1024
sliding_window: int = 128 # n_win, every layer
compress_rates = {"compressed_sparse_attention": 4, # m
"heavily_compressed_attention": 128} # m'
index_topk: int = 512 # compressed *entries*, not tokens
o_groups: int = 8
o_lora_rank: int = 1024
hc_mult: int = 4 # manifold-constrained hyper-connections
What replaced MLA
Side by side, V3.2 in the absorbed form that DSA actually runs, V4 with its compressor branch left out. Both columns are MQA — one shared KV entry per token, read by every head — which makes the remaining difference easy to see.
# ---- V3.2: MLA, absorbed form ------------------------------------------------
q = self.q_b_proj(self.q_a_layernorm(self.q_a_proj(x))) # [B, H, S, 128+64]
q_nope, q_rot = q.split([qk_nope_head_dim, qk_rope_head_dim], -1)
q_nope = einsum("bhsd,hdr->bhsr", q_nope, W_UK) # fold W_UK in: 128 -> 512
c_kv = self.kv_a_proj_with_mqa(x) # [B, S, 512+64]
c_kv, k_rot = c_kv.split([kv_lora_rank, qk_rope_head_dim], -1)
c_kv = self.kv_a_layernorm(c_kv) # [B, 1, S, 512] — the whole cache
q_rot, k_rot = apply_rotary_pos_emb_interleave(q_rot, k_rot, cos, sin)
s = q_nope @ c_kv.transpose(-1, -2) + q_rot @ k_rot.transpose(-1, -2) # key is 512 + 64
o = softmax(s) @ c_kv # value is 512 — K and V differ
o = einsum("bhsr,hdr->bhsd", o, W_UV) # unfold W_UV: 512 -> 128
return self.o_proj(o.reshape(B, S, -1)) # H*128 = 16384 -> 7168
# ---- V4: shared-KV MQA ------------------------------------------------------
q = self.q_b_norm(self.q_b_proj(self.q_a_norm(self.q_a_proj(x)))) # [B, H, S, 512]
q = apply_rotary_pos_emb(q, cos, sin) # partial RoPE, trailing 64
kv = self.kv_norm(self.kv_proj(x)) # [B, 1, S, 512], no up-projection
kv = apply_rotary_pos_emb(kv, cos, sin) # same partial RoPE
kv = past_key_values.update(kv, kv, self.layer_idx)[0] # sliding window, K == V
o = attention(q, kv, kv, s_aux=self.sinks) # K and V are the SAME tensor
o = apply_rotary_pos_emb(o, cos, -sin) # undo the rotation V carried
g = self.o_a_proj(o.reshape(B, S, o_groups, -1)).flatten(2) # 8 x (4096 -> 1024) = 8192
return self.o_b_proj(g) # 8192 -> 4096
The up-projections are gone. Absorbed MLA still carries $W^{UK}$ / $W^{UV}$, folded into the query and applied to the output, per head, $128 \leftrightarrow 512$. V4 deletes both — its head is natively $512$ wide, so there is nothing to reconstruct. That also collapses K and V into one tensor; absorbed MLA still attends with a $512{+}64$ key against a $512$ value.
Partial RoPE, and its inverse on the output. V3.2 scores the rope key with a second matmul; V4 rotates the trailing 64 dims of the one shared tensor, so one matmul covers both. That tensor is also the value, so the output inherits the rotation and V4 undoes it at the negative query position — apply_rotary_pos_emb(o, cos, -sin). Each entry’s contribution then depends on $R_{-t}R_s = R_{s-t}$, relative distance only.
Grouped low-rank output projection. This is what pays for the wide head. 64 heads × 512 stacks to 32768 (65536 on V4-Pro), and a direct $32768 \to 4096$ projection would be 134M params per layer. V4 splits the heads into 8 groups of 8, projects each group on its own, then mixes once:
grouped = attn_output.reshape(B, S, 8, -1) # [B, S, 8, 4096] 8 consecutive heads per group
grouped = self.o_a_proj(grouped) # [B, S, 8, 1024] block-diagonal, no cross-group
grouped = grouped.flatten(2) # [B, S, 8192]
output = self.o_b_proj(grouped) # [B, S, 4096]
o_a_proj subclasses nn.Linear, so the weight stays one [8192, 4096] tensor for checkpointing and TP; the forward reinterprets it as 8 blocks of [1024, 4096] and runs them as a single bmm batched over the group axis. $8 \times 4096 \times 1024 + 8192 \times 4096 = 67\text{M}$ — half the dense projection in both params and MACs, still two dense GEMMs.
One shape caveat: DeepseekV4DecoderLayer does not hand [B, S, D] to attention. V4’s residual is hc_mult = 4 parallel streams, [B, S, 4, D], carried that way through the model (mHC, manifold-constrained hyper-connections). Each block collapses them to one sequence on the way in and scatters the result back out afterwards, so attention still runs once per layer on an ordinary [B, S, D] — the four streams cost it nothing.
The sliding window
Every layer, whichever compressor it carries, sees only the last 128 tokens of the raw sequence. In training that is purely a mask — one is built for the whole model and handed to every layer, since all V4 layer types use the same window:
causal_mask = create_sliding_window_causal_mask(...) # [B, 1, S, S], banded
kv itself stays [B, 1, S, 512]; nothing is dropped, the band just hides anything older than 128 positions. (At inference the cache implements the same band by physically keeping only the last sliding_window - 1 entries.)
Attention sinks. A window this short needs an escape valve. self.sinks is one learned scalar per head, appended to each score row as an extra column before the softmax and dropped immediately after:
sinks = module.sinks.reshape(1, -1, 1, 1).expand(query.shape[0], -1, query.shape[-2], -1)
combined_logits = torch.cat([attn_weights, sinks], dim=-1) # [B, H, S, T] -> [B, H, S, T+1]
probs = F.softmax(combined_logits, dim=-1)
scores = probs[..., :-1] # the sink column has no value vector
The surviving weights sum to $1 - p_{\text{sink}}$, not $1$: a head may decline to attend and let its output shrink toward zero. A full-causal model gets this for free — heads park surplus attention on token 0 — but inside a 128-token window token 0 scrolled out long ago.
That it patches windowing rather than attention in general is clearest in MiMo-V2-Flash, which mixes both layer kinds and allocates the parameter accordingly:
self.sinks = nn.Parameter(torch.empty(num_attention_heads)) if is_swa else None
V4 gives every layer one because every layer is windowed. The code is gpt-oss’s eager_attention_forward, imported — but gpt-oss pairs the same window with alternating full-attention layers, and V4 has none at all.
HCA: sliding window + one compressor
Figure 4 of the V4 technical report. Two branches concatenated, no indexer anywhere: every compressed entry reaches attention. The grey box names the base attention — Shared Key-Value MQA, the K=V arrangement from the previous section.
An HCA layer is the sliding window plus one compressor and nothing else, so this is where the compressor is worth unpacking. All three compressors in V4 — HCA’s, CSA’s, and the one inside CSA’s indexer — are this same skeleton at different widths:
self.kv_proj = nn.Linear(4096, 512) # hidden_size -> head_dim
self.gate_proj = nn.Linear(4096, 512)
self.position_bias = nn.Parameter(torch.empty(128, 512)) # [m, head_dim]
self.kv_norm = DeepseekV4RMSNorm(512)
kv_proj / gate_proj | position_bias | entry width | |
|---|---|---|---|
| HCA | 4096 -> 512 | [128, 512] | 512 |
| CSA | 4096 -> 1024 | [4, 1024] | 512 |
| CSA’s indexer | 4096 -> 256 | [4, 256] | 128 |
The 512-wide entry matches the attention’s own kv, which is what lets it be concatenated onto the windowed keys. Collapsing the windows, training path — no cache, so a tail that does not fill a window is dropped:
kv = self.kv_proj(hidden_states) # [B, S, 4096] -> [B, S, 512]
gate = self.gate_proj(hidden_states) # [B, S, 4096] -> [B, S, 512]
usable = (S // m) * m # whole windows only
kv, gate = kv[:, :usable], gate[:, :usable]
kv = kv.view(B, n_win, m, 512) # [B, n_win, 128, 512]
gate = gate.view(B, n_win, m, 512) + self.position_bias # [m, 512] broadcasts
w = gate.softmax(dim=2) # over the m axis -> [B, n_win, 128, 512]
compressed = self.kv_norm((kv * w).sum(dim=2)) # [B, n_win, 512] one entry per window
positions = torch.arange(n_win) * m # first token of each window
compressed = apply_rotary_pos_emb(compressed.unsqueeze(1), cos, sin).squeeze(1)
Two things to read off that. softmax(dim=2) runs over the $m$ axis, so each of the 512 channels gets its own weights over the window; the product is elementwise and the sum is over slots, so channel $c$ of the entry sees only channel $c$ of the inputs. An entry is 512 independent convex combinations stacked, not an average — a mean would dilute a token that only one channel cares about by $m$. And RoPE lands at the window’s first absolute position $w\,m$, not at the entry’s index in the compressed sequence.
The dropped tail is at most $m-1$ tokens, and $m' = 128 =$ sliding_window, so those tokens are still visible verbatim through the window. The two branches overlap rather than partition.
On the attention side HCA adds nothing: no indexer, no selection, every query uses every causally legal entry. Entry $w$ pools tokens $[wm', (w+1)m')$, so query $t$ may use it only once that window has closed:
entry_indices = torch.arange(compressed_len)
causal_threshold = (position_ids + 1) // self.compress_rate # [B, S]
block_bias = compressed_kv.new_zeros((batch, 1, seq_len, compressed_len)) # [B, 1, S, E]
block_bias = block_bias.masked_fill(
entry_indices.view(1, 1, 1, -1) >= causal_threshold.unsqueeze(1).unsqueeze(-1),
float("-inf"),
)
At 128k an HCA layer holds $128\text{k}/128 = 1024$ entries — dense attention over 1024 keys plus a 128-token window needs no selection at all. HCA is the cheap global glue: heavy compression, full coverage.
CSA: two changes to HCA
Figure 3 of the same report. The left half is HCA’s picture with a Top-k Selector spliced into the compressed branch. The dashed box is worth a second look: the Lightning Indexer carries its own Token-Level Compressor, producing a separate set of compressed indexer keys at index_head_dim = 128 rather than scoring against the entries the attention will use. Two compressors run per CSA layer, which is what the "compressor" / "indexer" cache keys keep apart.
A CSA layer is an HCA layer with $m$ dropped from 128 to 4 and two changes on top. The skeleton, the RoPE bookkeeping and the causal threshold are unchanged.
(a) The windows overlap. The goal is an entry pooled from $2m = 8$ tokens but emitted every $m = 4$ — width 8, stride 4. Rather than gather overlapping spans, CSA keeps the non-overlapping view and doubles the channel width instead: each token emits two projections, Cb for its own window’s entry and Ca for the next one. Two slice-assignments lay them into a 2m-slot axis, and the same softmax(dim=2) does the rest:
kv = kv.view(B, n_win, 4, 1024) # [B, n_win, 4, 1024]
new_kv = kv.new_zeros((B, n_win, 8, 512)) # [B, n_win, 8, 512]
new_gate = gate.new_full((B, n_win, 8, 512), float("-inf"))
new_kv[:, :, 4:] = kv[..., 512:] # Cb of the current window
new_gate[:, :, 4:] = gate[..., 512:]
new_kv[:, 1:, :4] = kv[:, :-1, :, :512] # Ca of the previous window
new_gate[:, 1:, :4] = gate[:, :-1, :, :512]
# the last window's Ca has no next entry to feed and is dropped here; the cached
# path saves it and uses it as window 0's first half on the following call
Only the slot count changes, 4 to 8, and one softmax spans both halves so the two windows compete for the same mass. The cache still holds only $T/m$ entries, and shapes stay static — no gather, no strided view. Window 0 has no predecessor and its first half simply stays -inf, which softmax turns into exactly zero weight. The cost is one wider projection, 4096 -> 1024 instead of 4096 -> 512.
(b) The entries are selected, not all used. At $m=4$ a 128k context leaves 32k entries; HCA attends densely over its 1024, CSA cannot. So V4 puts the Part 1 indexer back one level up, scoring compressed entries instead of tokens — at the cost of a third compressor, since it builds its own keys with its own copy of the Ca/Cb skeleton at index_head_dim = 128 rather than reusing the entries attention will consume.
q = self.q_b_proj(q_residual).view(B, S, 64, 128) # [B, S, 64, 128] same q_residual as MLA
scores = torch.matmul(q.float(), compressed_kv.transpose(-1,-2).unsqueeze(1)) # [B, S, 64, E]
scores = F.relu(scores) * self.softmax_scale
weights = self.weights_proj(hidden_states).float() # [B, S, 64]
index_scores = (scores * weights.unsqueeze(-1)).sum(dim=2) # [B, S, E]
Structurally identical to V3.2’s scorer, head axis summed away before the top-$k$. index_topk = 512 entries × $m = 4$ gives 2048 tokens of coverage — V3.2’s budget at a quarter of the scoring cost.
One wrinkle DSA did not have: early queries have fewer than 512 legal entries, so topk returns junk. It is marked with a -1 sentinel and dropped by scattering into a spare column that gets sliced off:
top_k_indices = index_scores.masked_fill(future_mask, float("-inf")).topk(512, dim=-1).indices
top_k_indices = torch.where(top_k_indices >= causal_threshold.unsqueeze(-1), -1, top_k_indices)
safe = torch.where(top_k_indices >= 0, top_k_indices, compressed_len)
block_bias = compressed_kv.new_full((B, 1, S, compressed_len + 1), float("-inf"))
block_bias.scatter_(-1, safe.unsqueeze(1), 0.0)
return compressed_kv, block_bias[..., :compressed_len] # [B, 1, S, E]
The layer schedule
With the pieces in hand, here is how a 43-layer model arranges them. The per-layer type comes from config.layer_types:
interleave = ["compressed_sparse_attention" if i % 2 else "heavily_compressed_attention"
for i in range(max(n - 2, 0))]
self.layer_types = ["heavily_compressed_attention"] * min(n, 2) + interleave
Two HCA layers to bootstrap, then strict alternation. Every layer also runs the sliding window, and assembling the two branches is the last thing the attention forward does before scoring:
if self.compressor is not None: # CSA and HCA layers only
compressed_kv, block_bias = self.compressor(hidden_states, q_residual, position_ids,
past_key_values, self.layer_idx)
kv = torch.cat([kv, compressed_kv], dim=2) # local window ‖ compressed history
self.compressor is the HCA or CSA module from the previous two sections; block_bias is the per-query mask it returns.
Side by Side
| V3.2-Exp (DSA) | V4 (CSA layers) | V4 (HCA layers) | |
|---|---|---|---|
| Base attention | MLA, 128 heads, latent 512 | shared-KV MQA, 64 heads × 512, K=V | same |
| Local branch | — (top-k covers everything) | 128-token sliding window | 128-token sliding window |
| Long-range branch | top-2048 tokens from full cache | top-512 compressed blocks, $m=4$ | all compressed blocks, $m'=128$ |
| Compression | none — dense cache | gated pooling, overlapping ($2m$ wide, stride $m$) | gated pooling, non-overlapping |
| Indexer | yes, over tokens | yes, over compressed blocks | no |
| Cache/token/layer | 576 (MLA) + 128 (indexer) | 512 × 128 window + 512/4 + 128/4 amortized | 512 × 128 window + 512/128 amortized |
| Long-context cache growth | $O(T)$ | $O(T/4)$ | $O(T/128)$ |
The single-sentence version: V3.2 made the attention read sparse; V4 made the memory itself sparse, and then spent the savings on a much larger head dim (512 vs 128) and a bigger local window.
Caveats
- Everything above is the transformers reference implementation: correct semantics, deliberately not the fast path — both models materialize additive masks where the production kernels consume indices, and V3.2 ships
_supports_flash_attn = Falsebecause the sparse kernels aren’t wired up here yet. Don’t read FLOPs off this code. - V3.2’s indexer skips the Hadamard rotation and FP8 quantization of DeepSeek’s own kernel — mathematically equivalent for the dot products, but not bit-identical.
- V4’s
layer_typesalso admits a plain"sliding_attention"entry (no compressor at all), which the released schedules don’t use but the code supports. - V4’s other two departures — mHC (the four-stream residual above, mixed by a Sinkhorn-projected doubly-stochastic matrix) and the MoE changes (
sqrtsoftplusrouting, group-limited routing dropped, Hash-MoE replacing the dense bootstrap layers) — are out of scope and get their own post.
References
src/transformers/models/deepseek_v32/modular_deepseek_v32.pysrc/transformers/models/deepseek_v4/modular_deepseek_v4.py,configuration_deepseek_v4.py- DeepSeek-V3.2 technical report, arXiv:2512.02556 — DSA and its two-stage training recipe
- DeepSeek-V4 technical report, arXiv:2606.19348 — §2.3.1 (CSA), §2.3.2 (HCA), §2.2 (mHC)
- Related: Absorbed MLA vs. Naive MLA