Yunsheng Ni

One Formula, Two Jobs: How RoPE and Timestep Embedding Are Built

RoPE and diffusion timestep embedding share the same sinusoidal skeleton and even the same frequency table. This note builds both from angular frequency up, and explains what base / max_period actually control.

Absorbed MLA vs. Naive MLA: Same Linear FLOPs, 3.4x Core Attention

Matrix absorption rewrites MLA to attend directly on the compressed latent. A side-by-side PyTorch implementation plus a FLOPs breakdown shows the token-linear cost is bit-for-bit identical while core attention gets 3.4x more expensive — which is exactly why dense causal training keeps the naive form and sparse attention (DSA) training uses the absorbed one.

Progressive CUDA GEMM Optimization: From Memory-Bound to Swizzling

A step-by-step guide to optimizing FP32 CUDA GEMM kernels. Learn how to overcome warp-level memory coalescing bottlenecks, eliminate 32-way shared memory bank conflicts using memory padding, and implement zero-waste XOR address swizzling.

Loss Reduction in Distributed Training

An analysis of how Data Parallelism (DP) and Context Parallelism (CP) affect loss reduction, and how to maintain mathematical equivalence between distributed and single-device training for LLM and Video DiT models.

Computing Global Gradient Norm in Distributed Training: TP, DP_Shard, DP_Replicate, EP, and PP

A mathematical and engineering guide to calculating the exact global gradient norm across complex hybrid parallel training topologies. This post details the required hierarchical synchronization sequence across TP, DP, EP, and PP process groups to compute the norm without materializing full tensors, preventing double-counting and OOM errors.

Demystifying FlashAttention: Forward, Backward, and Triton Implementation

A breakdown of FlashAttention’s forward and backward passes, including Online Softmax, LogSumExp materialization, gradient recomputation, and core Triton implementations.

The Devil in the Details: Engineering Tricks for SOTA Video Models

Theory is clean, but training is messy. This note covers 5 essential engineering tricks—from Timestep Shifting to 3D RoPE—that stabilize training and boost performance.

Deep Dive into Triton GEMM Optimization: From Naive Tiling to Hopper TMA

A step-by-step guide to optimizing GEMM in Triton, covering Tiling, Autotuning, L2 Cache Optimizations, and Hopper TMA.

Roofline Analysis of LLMs on H200: Performance Modeling and Recomputation Strategies

A quantitative Roofline analysis of LLMs on NVIDIA H200. We derive compute-bound thresholds, analyze the 1:10 communication bottleneck, and propose optimal strategies for activation recomputation and operator fusion to maximize hardware efficiency.

From DDPM to Flow Matching: The Evolution of Generative Trajectories

A technical note on the shift from noise prediction (DDPM) to velocity prediction (Flow Matching), and how CFG acts as a vector field modifier.