Qwen3.8-27B-NVFP4 1M context. So far so good.

I am a beginner, Took a while to get started, get everything right. This setup is native not container. Still not sure if I did this right, or if I can tune this more. Environment=HF_HUB_OFFLINE=1 Environment=VLLM_LOGGING_LEVEL=INFO Environment=VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 Environment=PATH=/home/suryakiranc/vllm/.venv/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin Environment=CUDA_HOME=/usr/local/cuda ExecStart=/home/suryakiranc/vllm/.venv/bin/vllm serve unsloth/Qwen3.8-27B-NVFP4 --served-model-name unsloth/Qwen3.8-27B-NVFP4 --safetensors_load_strategy prefetch --tensor-parallel-size 4 --reasoning-parser qwen3 --tool-call-parser qwen3_xml --enable-auto-tool-choice --gpu-memory-utilization 0.91 --kv-cache-dtype fp8 --max-num-batched-tokens 16384 --speculative-config '{"method": "mtp", "num_speculative_tokens": 3}' --mm-encoder-tp-mode data --hf-overrides '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}' --max-model-len 1000000 --host 0.0.0.0 --port 8000 submitted by /u/No_Night679 [link] [comments]

Related

• I'm really hoping we're in 2026's 2-month-gap between QwQ and Qwen3 right now
• Qwen3.8-Flash-Next at 170K context on a single 96 GB card. ~110 tok/s.

Source: r/LocalLLaMA | 2026-09-15