Hardware
🧵 Slime: The Most Elegant & Comfortable RL Training Framework Ever A deep dive into why Slime redefines LLM RL training with clean architec…
🧵 Slime: The Most Elegant & Comfortable RL Training Framework Ever A deep dive into why Slime redefines LLM RL training with clean architecture & production-grade engineering ✨ Insights from Zhihu con
🧵 Slime: The Most Elegant & Comfortable RL Training Framework Ever A deep dive into why Slime redefines LLM RL training with clean architecture & production-grade engineering ✨ Insights from Zhihu contributor Xavier 📌 What Is Slime In One Sentence? Slime is a streamlined RL training framework built on SGLang (Inference) + Megatron (Training) + Ray (Orchestration).It’s not just a simple stack—it stitches top-tier open-source projects together with perfectly polished interfaces.Core design philosophy: Fully decouple training & inference, connected via streamlined data flow. Compared to veRL / OpenRLHF: ✅ Native SGLang backend → high concurrency, continuous batching, prefix caching (no messy vLLM wrapper) ✅ Native Megatron backend → full TP/PP/EP/CP parallelism, seamless MoE training ✅ Lightweight Ray scheduling → Placement Group + Remote Actor (no bloated Ray Train) 🏗️ Global Architecture: 3 Modules, One Pipeline 🖥️ Ray Cluster Core Workflow:Data Buffer (Prompt Manager → Buffer & Filter)↔️ Rollout (SGLang → Sampling + RM Scoring + Filtering)↔️ Training (Megatron → Actor/Critic + PPO/GRPO) 🔁 Simplified Core Training Loop 1.Allocate GPU resources via Placement Group 2.Launch SGLang rollout engine 3.Initialize Megatron Actor/Critic models 4.Sync initial weights to SGLang 5.Repeat 3-beat cycle: Generate (SGLang) → Train (Megatron) → Sync Weights 🎯Elegance = ultra-simple top-level logic, all complexity encapsulated inside modules 🎛️ 4 Core Design Flexibilities ⚙️ Resource Scheduling: Colocate (shared GPU) / Disaggregate (separate GPU pools) 🔄 Training Mode: Synchronous / Asynchronous training 🧪 Sampling Logic: Standard sampling / Over-sampling / Multi-turn tool calling 🤖 Model Type: Dense / MoE, full tensor/pipeline/context parallel support 🔧 Plug & Play Customization (All Extensible) Slime lets you customize every component via CLI params—no need to fork the repo 🛠️ Key Customization Points ✅ Custom Reward Model: Write an async func to define your own reward logic (easiest entry) ✅ Custom Generate Func: Control multi-turn dialogue, tool calling & external API integration ✅ Custom Rollout Func: Fully take over sampling concurrency & filtering logic ✅ Custom DataSource: Fetch prompts from API / local files / dynamic data streams ✅ Dynamic Filter: Discard low-value sample groups (e.g., zero-variance GRPO samples) ✅ Custom Loss Function: Rewrite PPO/GRPO loss calculation freely All custom code loads dynamically via --custom-xxx-path config 📝 🚀 Ray GPU Scheduling Magic Two deployment modes for all cluster scales: 🔹 Colocate Mode: Train & inference share GPUs → high utilization, ideal for small 8-card servers 🔹 Disaggregate Mode: Independent GPU pools → train-infer overlap, perfect for multi-node clusters Slime stabilizes Ray Placement Group GPU mapping via IP/GPU ID sorting to guarantee reproducibility 🔒 ⚡ SGLang Rollout Engine Internals 3-layer abstraction:RolloutManager → RolloutServer → ServerGroup → SGLangEngine Standout design highlights: 🔸 Over-sampling + Dynamic Filter: Pre-sample extra data, filter invalid groups on the fly 🔸 Async Concurrent Sampling: Process completed groups immediately with FIRST_COMPLETED 🔸 Abort Mechanism: Stop redundant sampling once target data size is met, save compute 🔸 Singleton GenerateState: One-time tokenizer & connection initialization 🧠 Megatron Training Backend Native support for mainstream RL algorithms: ✅ GRPO: No Critic needed, group-wise reward normalization (most popular) ✅ PPO: Classic Actor-Critic with GAE advantage estimation ✅ REINFORCE++: Token-level baseline optimization Seamless support for Dense & large MoE models with full parallelism 📊 🔄 Weight Sync: The Hard Engineering Solved Two high-performance sync paths: 🔹 Colocate: IPC + Gloo → intra-node low-latency weight transfer 🔹 Disaggregate: NCCL Broadcast → cross-node distributed sync MoE OOM prevention: Chunked Bucket Weight Update → sync parameters in small batches, release memory instantly 🧩 💡 Core Takeaways ✨ Slime’s elegance lies in integrating mature top-tier stacks with clean decoupled design ✨ Minimal top-level logic, maximal internal engineering depth ✨ Fully pluggable customization for all RL scenarios (Math / Code / Agent / MoE) ✨ Optimized for both small single-node & large multi-node clusters 🔗Full article：https://zhuanlan.zhihu.com/p/2035357069637890331 #LLM #RLTraining #SGLang #AIInfrastructure #MoE #MachineLearning
Source: Zhipu AI (X) | 2026-05-11