Parallelism on Yunsheng Nihttps://niyunsheng.github.io/tags/parallelism/Recent content in Parallelism on Yunsheng NiHugo -- 0.154.5en-usContent is licensed under CC BY-NC-SA 4.0.Sat, 21 Mar 2026 15:41:55 +0800Loss Reduction in Distributed Traininghttps://niyunsheng.github.io/loss-reduction-dp-cp/Sat, 21 Mar 2026 15:41:55 +0800https://niyunsheng.github.io/loss-reduction-dp-cp/An analysis of how Data Parallelism (DP) and Context Parallelism (CP) affect loss reduction, and how to maintain mathematical equivalence between distributed and single-device training for LLM and Video DiT models.Computing Global Gradient Norm in Distributed Training: TP, DP_Shard, DP_Replicate, EP, and PPhttps://niyunsheng.github.io/global-grad-norm-hybrid-parallel/Sat, 21 Mar 2026 07:02:02 +0800https://niyunsheng.github.io/global-grad-norm-hybrid-parallel/A mathematical and engineering guide to calculating the exact global gradient norm across complex hybrid parallel training topologies. This post details the required hierarchical synchronization sequence across TP, DP, EP, and PP process groups to compute the norm without materializing full tensors, preventing double-counting and OOM errors.