How to automatically find the batch size when using Accelerate with FSDP2? [D]

Hi, For single-GPU training, I’m using Hugging Face SFTTrainer with auto_find_batch_size=True, which automatically reduces the batch size after a CUDA OOM until it finds a batch size that works. I would like to have similar behavior when training on multiple GPUs on a single node using accelerate launch with FSDP2. Is there a supported way to automatically determine or reduce the batch size when using Accelerate + FSDP2? In particular, I’m wondering how this should be handled when one of the distributed processes encounters a CUDA OOM. Can Accelerate restart the distributed training with a smaller batch size, or does this need to be implemented externally? If this is not well supported with FSDP2, what multi-GPU training approach would you recommend if automatic batch-size detection/recovery is an important requirement? Thanks! submitted by /u/zdeneklapes [link] [comments]

Source: r/MachineLearning | 2026-09-14