Model Releases
RAM Offloading with vLLM - tcclaviger appreciation post
Thanks to tcclaviger, vLLM now has expert RAM offloading support (link). This makes frontier models much more accessible on a local setup! I was able to run the original DeepSeek-V4-Flash-Vision-Exp o
Thanks to tcclaviger, vLLM now has expert RAM offloading support (link). This makes frontier models much more accessible on a local setup! I was able to run the original DeepSeek-V4-Flash-Vision-Exp on four R9700s. podman run --rm -it --init --network host --ulimit memlock=-1:-1 -v /models:/models:ro -v ~/.vllm-cache:/cache -e VLLM_ROCM_USE_AITER=0 -e ROCR_VISIBLE_DEVICES=0,1,2,3 -e VLLM_CACHE_ROOT=/cache/vllm -e TORCHINDUCTOR_CACHE_DIR=/cache/inductor -e TRITON_CACHE_DIR=/cache/triton --device /dev/kfd --device /dev/dri --group-add keep-groups --annotation run.oci.keep_original_groups=1 --security-opt label=disable --security-opt seccomp=unconfined --shm-size 160g docker.io/tcclaviger/vllm@sha256:ef99b3d07c3f15e7978528c7510762ba024df9ab4242070d8ed092cd4cc1a694 /models/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp --served-model-name DeepSeek-V4-Flash-Vision-Exp --tensor-parallel-size 4 --enable-expert-offload --expert-offload-mem 160 --reasoning-parser deepseek_v4 --reasoning-config '{"reasoning_parser":"deepseek_v4","reasoning_start_str":"","reasoning_end_str":""}' --tool-call-parser deepseek_v4 --enable-auto-tool-choice --max-num-seqs 8 --enable-prefix-caching --enable-chunked-prefill --max-num-batched-tokens 2048 --kv-cache-dtype fp8 --block-size 256 --max-model-len 256000 --gpu-memory-utilization 0.97 --mm-processor-cache-gb 4.0 --override-generation-config '{"max_tokens": 128000, "temperature": 1.0, "top_p": 0.95}' --speculative-config '{"method":"dspark","model":"/models/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp","num_speculative_tokens":3,"draft_sample_method":"probabilistic","enable_adaptive_verification":false}' --compilation-config '{"cudagraph_capture_sizes": [4,8,12,16], "max_cudagraph_capture_size": 16}' --host 0.0.0.0 --port 8090 submitted by /u/sloptimizer [link] [comments]
Related
- [[deepseek-v4-flash-0731-full-1m-context-on-a-single-rtx5090-d|[Deepseek-V4-Flash-0731] Full 1M context on a single RTX5090 + DDR5 Desktop Setup with VLLM CPU/Ram Offloading, ~800 tps pp & 15+ tps decode [Agentic Coding]]]
- DeepSeek-V4-Flash-Vision-Exp (285B MoE) on 10-12x RTX 3090 — spec decoding, vision
- Anyone running DeepSeek-V4-Flash-0731 on MI325X with vLLM? Mine is behaving completely broken
Source: r/LocalLLaMA | 2026-09-29