BASP: Communication-Efficient Batch-Aware Sequence Parallelism for LLM Training

arXiv:2609.03151v1 Announce Type: cross Abstract: Long-context reasoning for large language models (LLMs) is becoming increasingly important, but training over long sequences remains challenging due to massive memory and communication requirements. Sequence parallelism has emerged as an essential technique for addressing bottlenecks in long sequence LLM training. However, we observe that existing sequence parallelism methods are batch-agnostic and apply uniform sequence partitioning across all batch sizes, resulting in inefficient communication. In this paper, we introduce Batch- Aware Sequence Parallelism (BASP), a sequence parallelism approach that leverages batch structure to reduce communication overhead. BASP exploits batch structure by partitioning GPUs into disjoint sequence-parallel groups according to the micro- batch size. This design reduces the all-to-all communication group size, thereby localizing communication and improving training efficiency. Experimental results on an NVIDIA A100 cluster show that BASP improves end-to-end training time by up to 1.17 - 1.31x in Llama and Qwen models compared to standard sequence parallel baselines, while preserving identical model accuracy and memory usage.

Related

• AutoSP: Unlocking Long-Context LLM Training Via Compiler-Based Sequence Parallelism
• A Tabular Schedule Abstraction for Communication-Aware Evaluation of Pipeline-Parallel LLM Training
• Understanding and Improving Communication Performance in Multi-node LLM Inference
• Protocol Models: Scaling Decentralized Training with Communication-Efficient Model Parallelism

Source: arXiv cs.LG | 2026-09-04