Model Releases
Llama-CPP Parallel Agents --> fine for decode, but one agent's prefill will grind all other agents to a halt
Testing with 3-5 agents. Decode performance is superb, however if one performs a web search and needs to process a few thousand tokens, ALL other agents will grind to a halt: I've tried tuning a littl
Testing with 3-5 agents. Decode performance is superb, however if one performs a web search and needs to process a few thousand tokens, ALL other agents will grind to a halt: I've tried tuning a little bit, but no luck. example command of mine (this server is ONLY used for the sub-agents): ./llama-server --model /models/Gemma4-26B/gemma-4-26B-A4B-it-UD-Q5_K_M.gguf --model-draft /models/Gemma4-26B/mtp-gemma-4-26B-A4B-it-Q8_0.gguf --device Vulkan0 --device-draft Vulkan0 --split-mode none --main-gpu 0 --gpu-layers all --spec-type draft-mtp --spec-draft-n-max 3 --ctx-size 240000 --parallel 3 --batch-size 2048 --ubatch-size 512 --flash-attn on --kv-unified --cache-reuse 256 --host 0.0.0.0 --port 8081 I'm fairly new to parallel agents. Any thoughts/suggestions on what i should be doing differently? submitted by /u/EmPips [link] [comments]
Related
- GLM-5.2 UD-IQ1_M on llama.cpp — 5090 + 3090 Ti speed test (~ 579 t/s prefill @ 8k ctx, ~324 t/s prefill @ 57k ctx, ~10.6 t/s decode)
- kv-cache : avoid kv cells copies by ggerganov · Pull Request #24277 · ggml-org/llama.cpp
- Ran DS V4-Flash-0731 Locally on 3xMI50 32GB @ ~15 t/s TG
- I updated my localy run benchmark with DeepSeek V4 Flash 0731
Source: r/LocalLLaMA | 2026-08-11