Model Releases
SkewAdam: A tiered optimizer that cuts MoE state memory by 97% (fits a 6.7B MoE on a 40GB GPU) [R]
Paper:https://arxiv.org/abs/2607.19058 Code (GitHub):https://github.com/nuemaan/skewadam Hi everyone, I just published a preprint on a new optimizer designed to tackle the massive VRAM bottleneck in M
Paper:https://arxiv.org/abs/2607.19058 Code (GitHub):https://github.com/nuemaan/skewadam Hi everyone, I just published a preprint on a new optimizer designed to tackle the massive VRAM bottleneck in Mixture-of-Experts (MoE) training. If you've trained MoEs, you know that optimizer state is usually the largest single line item in the memory budget. AdamW, for example, spends 50.6 GB of state memory just to update a 12.6 GB model. I built SkewAdam to fix this by using a tiered state allocation. Instead of treating all parameters equally, it allocates precision based on parameter behavior: Backbone (5% of params): Momentum + Factored 2nd moment Experts (95% of params): Factored 2nd moment only Router (<0.01% of params): Exact 2nd moment The Hardware Results: Optimizer state memory drops from 50.6 GB to 1.29 GB (a 97.4% reduction). Peak training memory drops from 81.4 GB to 31.3 GB. This allows a 6.78B MoE to fit comfortably on a single 40GB GPU without sacrificing convergence or router stability. submitted by /u/Kooky-Ad-4124 [link] [comments]
Related
- SAGE: Sign-Adaptive Gradient for Memory-Efficient LLM Optimization
- KIV: 1M token context window on a RTX 4070 (12GB VRAM), no retraining, drop-in HuggingFace cache replacement - Works with any model that uses DynamicCache [P]
- MEMTIER: Tiered Memory Architecture and Retrieval Bottleneck Analysis for Long-Running Autonomous AI Agents
- I released a softmax-free attention model at GPT-2 Medium scale (~354M params, 11.5B tokens): structural sparsity + tile-skipping kernels for long-context VRAM savings. Open weights + custom Triton kernels [R]
Source: r/MachineLearning | 2026-07-22