System Optimization

Matrix absorption rewrites MLA to attend directly on the compressed latent. A side-by-side PyTorch implementation plus a FLOPs breakdown shows the token-linear cost is bit-for-bit identical while core attention gets 3.4x more expensive — which is exactly why dense causal training keeps the naive form and sparse attention (DSA) training uses the absorbed one.

A quantitative Roofline analysis of LLMs on NVIDIA H200. We derive compute-bound thresholds, analyze the 1:10 communication bottleneck, and propose optimal strategies for activation recomputation and operator fusion to maximize hardware efficiency.

A rigorous breakdown of FLOPs in Llama-style architectures, deriving the relationship between linear projections and quadratic attention overhead, with insights into sample packing efficiency.