Archive

2026 ¹⁴

August ²

One Formula, Two Jobs: How RoPE and Timestep Embedding Are Built

Video Generation Video Generation Theory #RoPE #Timestep Embedding #DiT

August 6, 2026 · 15 min · 2989 words · Yunsheng Ni

Absorbed MLA vs. Naive MLA: Same Linear FLOPs, 3.4x Core Attention

System Optimization #MLA #DeepSeek #KV Cache #FLOPs #Roofline #Sparse Attention

August 2, 2026 · 18 min · 3665 words · Yunsheng Ni

March ⁴

Progressive CUDA GEMM Optimization: From Memory-Bound to Swizzling

Kernels GEMM Optimization #CUDA #GEMM #Shared Memory #Bank Conflict #Swizzling

March 22, 2026 · 11 min · 2155 words · Yunsheng Ni

Loss Reduction in Distributed Training

Distributed System #Parallelism #Context Parallelism

March 21, 2026 · 4 min · 809 words · Yunsheng Ni

Computing Global Gradient Norm in Distributed Training: TP, DP_Shard, DP_Replicate, EP, and PP

Distributed System #Parallelism #Gradient Clipping #FSDP

March 21, 2026 · 5 min · 900 words · Yunsheng Ni

Demystifying FlashAttention: Forward, Backward, and Triton Implementation

Kernels #Triton #Flash Attention

March 15, 2026 · 11 min · 2338 words · Yunsheng Ni

February ⁵

The Devil in the Details: Engineering Tricks for SOTA Video Models

Video Generation Video Generation Theory #DiT #Flow Matching #RoPE #Training Stability

February 25, 2026 · 6 min · 1211 words · Yunsheng Ni

Deep Dive into Triton GEMM Optimization: From Naive Tiling to Hopper TMA

Kernels GEMM Optimization #Triton #GEMM #Tiling #TMA #Swizzling

February 11, 2026 · 15 min · 3019 words · Yunsheng Ni

Roofline Analysis of LLMs on H200: Performance Modeling and Recomputation Strategies

System Optimization Performance Modeling #MFU #FLOPs #Roofline #H200 #Recomputation

February 9, 2026 · 6 min · 1217 words · Yunsheng Ni

From DDPM to Flow Matching: The Evolution of Generative Trajectories

Video Generation Video Generation Theory #DiT #DDPM #Flow Matching #Classifier-Free Guidance

February 3, 2026 · 5 min · 1032 words · Yunsheng Ni

From DiT to Hunyuan: The Evolution of adaLN-Zero in Generative Models

Video Generation Video Generation Theory Open Model Architectures #DiT #DDPM #Flow Matching

February 2, 2026 · 3 min · 507 words · Yunsheng Ni

January ³

Beyond Theoretical FLOPs: Analyzing MFU, HFU, and Attention Overhead in Transformers

System Optimization Performance Modeling #MFU #FLOPs #GEMM #Flash Attention

January 31, 2026 · 7 min · 1285 words · Yunsheng Ni

Visualizing 3D Attention: Bridging the Gap Between 1D Sequences and 3D Space

Video Generation 3D Sparse Attention #Sparse Attention #Visualization

January 26, 2026 · 2 min · 403 words · Yunsheng Ni

GPU & Network Constants

Hardware Performance Modeling #GPU Specs #H200 #Interconnect

January 25, 2026 · 2 min · 238 words · Yunsheng Ni

2026 14

August 2