Local Ai
Prevent CUDA OOM in PyTorch with dynamic lane switching
I built MEM v3 to solve a frustrating problem in PyTorch: CUDA Out-of-Memory crashes during long training and fine-tuning runs. Instead of restarting when memory spikes or keeping batch sizes overly s
I built MEM v3 to solve a frustrating problem in PyTorch: CUDA Out-of-Memory crashes during long training and fine-tuning runs. Instead of restarting when memory spikes or keeping batch sizes overly small just to be safe, MEM acts as a memory governor. It watches VRAM and throughput in real-time, then dynamically adjusts batch size and gradient accumulation on the fly without stopping the process. What it does: - Dynamic lane switching: Scales batch size up or down in milliseconds based on actual GPU memory pressure. - Chaos resistance: Tested against sudden +10 GB VRAM allocation shocks without crashing. - Crash-proof checkpoints: Uses atomic file replacement with SHA-256 checks across rotating slots, so power outages won't corrupt saved weights. - Live telemetry: Built-in local web dashboard to track loss, throughput, and lane switches. You can test it directly on a free Colab GPU without setting anything up locally: https://colab.research.google.com/github/nobazzy/mem-llm-orchestrator/blob/main/notebooks/mem_orchestrator_interactive_demo.ipynb Repo: https://github.com/nobazzy/mem-llm-orchestrator Would love to hear your thoughts and feedback! submitted by /u/uBazzyZ- [link] [comments]
Related
- R9V Update: now ~100 tok/s in TG on Qwen3.8 Flash Next IQ4_XS on x2 R9700 + 128GB RAM. Fixed crashes with n-gram SSD streaming, improved diagnostics, plus pinned images. Q4_K_XL now supported, 50 tok/s TG.
- 2× Radeon R9700 for Local AI Was Choosing AMD Instead of NVIDIA a Mistake Without CUDA?
Source: r/LocalLLaMA | 2026-09-27