Archive

2026  12

March  4

Progressive CUDA GEMM Optimization: From Memory-Bound to Swizzling

March 22, 2026 · 10 min · 2080 words · Yunsheng Ni

Loss Reduction in Distributed Training

March 21, 2026 · 4 min · 811 words · Yunsheng Ni

Computing Global Gradient Norm in Distributed Training: TP, DP_Shard, DP_Replicate, EP, and PP

March 21, 2026 · 5 min · 900 words · Yunsheng Ni

Demystifying FlashAttention: Forward, Backward, and Triton Implementation

March 15, 2026 · 12 min · 2348 words · Yunsheng Ni

February  5

The Devil in the Details: Engineering Tricks for SOTA Video Models

February 25, 2026 · 6 min · 1213 words · Yunsheng Ni

Deep Dive into Triton GEMM Optimization: From Naive Tiling to Hopper TMA

February 11, 2026 · 15 min · 3019 words · Yunsheng Ni

Roofline Analysis of LLMs on H200: Performance Modeling and Recomputation Strategies

February 9, 2026 · 6 min · 1221 words · Yunsheng Ni

From DDPM to Flow Matching: The Evolution of Generative Trajectories

February 3, 2026 · 5 min · 1035 words · Yunsheng Ni

From DiT to Hunyuan: The Evolution of adaLN-Zero in Generative Models

February 2, 2026 · 3 min · 507 words · Yunsheng Ni

January  3

Beyond Theoretical FLOPs: Analyzing MFU, HFU, and Attention Overhead in Transformers

January 31, 2026 · 7 min · 1298 words · Yunsheng Ni

Visualizing 3D Attention: Bridging the Gap Between 1D Sequences and 3D Space

January 26, 2026 · 2 min · 404 words · Yunsheng Ni

GPU & Network Constants

January 25, 2026 · 2 min · 238 words · Yunsheng Ni