Model Releases
Tensor parallel over RPC in my Ollama fork: two DGX Sparks now beat one (gpt-oss-120b 42.7 → 58.1 tok/s)
Follow-up to my fork that lets one Ollama use two DGX Sparks (128 GB each, direct 200 Gb/s cable, RDMA). Splitting a model by layers over llama.cpp RPC only bought capacity: the machines compute one a
Follow-up to my fork that lets one Ollama use two DGX Sparks (128 GB each, direct 200 Gb/s cable, RDMA). Splitting a model by layers over llama.cpp RPC only bought capacity: the machines compute one after the other, so a split model was always slower than the same model on one box. llama.cpp merged tensor parallel over RPC in October (PR #26610): two RPC servers each hold half of every layer and do the all-reduce between themselves, the client only drives them. I wired it into the fork. Ollama now starts a second RPC server for its own GPU next to llama-server and runs --rpc worker,local -dev RPC0,RPC1 -sm tensor -ts 1,1. Measured through Ollama (ollama cluster bench, context 32k, tg128, warm, mean of 3): Model | 1 node | layer split | tensor parallel qwen3.8:27b (with MTP draft) | 24.4 | 21.4 | 34.7 gpt-oss:120b | 42.7 | 33.8 | 58.1 DeepSeek-V4-Flash MXFP4, 156 GB | doesn't fit | 14.5 | 21.4Model | 1 node | layer split | tensor parallel qwen3.8:27b (with MTP draft) | 24.4 | 21.4 | 34.7 gpt-oss:120b | 42.7 | 33.8 | 58.1 DeepSeek-V4-Flash MXFP4, 156 GB | doesn't fit | 14.5 | 21.4 Prompt processing stays at about the layer-split level (~10% below one node for models that fit). Load time is a few seconds longer than the layer split. For reference: vLLM with TP=2 gave 55.7 tok/s for gpt-oss-120b on the same pair when I measured it a week ago (eager mode, single request). So llama.cpp RPC is now in the same range, while Ollama keeps loading and unloading models on demand. Limits: exactly two machines (llama.cpp pairs two RPC servers), one tensor model at a time, and the first server listens for the second on port + 1000 while the model loads. submitted by /u/ShadoByBB [link] [comments]
Related
- Ollama full GPU offload:
LLAMA_ARG_THREADS=1drastically reduced CPU usage with no performance penalty - Ctx size in Ollama Vs llama.cpp
- Testing Llama 3 locally: the model wasn’t the issue, the workflow was
Source: r/ollama | 2026-10-18