FYI You dont need expensive networking for multi-node gpu. 30t/s laguna Q2_K_XL (39.7GB) on 2x4060+1x4060 using a $20 usb->ethernet.

Turns out a regular ethernet cable between 2 nodes can run laguna UD-Q2_K_XL (39.7GB) using a direct point to point network. Interestingly on nvidia-smi dmon -s pucvmet -d 2, the inter/intra gpu traffic is not really capped in this setup - Uses ~30-70MB/s at peak # gpu pwr gtemp mtemp sm mem enc dec jpg ofa mclk pclk pviol tviol fb bar1 ccpm sbecc dbecc pci rxpci txpci # Idx W C C % % % % % % MHz MHz % bool MB MB MB errs errs errs MB/s MB/s 0 46 47 - 24 21 0 0 0 0 8751 2610 0 0 14557 4 0 - - 0 0 36 0 46 48 - 19 16 0 0 0 0 8751 2610 0 0 14557 4 0 - - 0 12 31 0 46 48 - 19 16 0 0 0 0 8751 2610 0 0 14557 4 0 - - 0 61 3 Benchmarks for 11k token prompt, 100k context: ubatch-size = 768 [58055] 2.53.851.946 I slot print_timing: id 0 | task 0 | prompt eval time = 19413.33 ms / 11719 tokens ( 1.66 ms per token, 603.66 tokens per second) [58055] 2.53.851.950 I slot print_timing: id 0 | task 0 | eval time = 125051.84 ms / 3536 tokens ( 35.37 ms per token, 28.28 tokens per second) [58055] 2.53.851.950 I slot print_timing: id 0 | task 0 | total time = 144465.17 ms / 15255 tokens [58055] 2.53.851.954 I slot print_timing: id 0 | task 0 | graphs reused = 3521 ubatch-size = 896 [44325] 2.25.234.581 I slot print_timing: id 0 | task 0 | prompt eval time = 19977.35 ms / 11719 tokens ( 1.70 ms per token, 586.61 tokens per second) [44325] 2.25.234.584 I slot print_timing: id 0 | task 0 | eval time = 95714.94 ms / 2124 tokens ( 45.06 ms per token, 22.19 tokens per second) [44325] 2.25.234.585 I slot print_timing: id 0 | task 0 | total time = 115692.29 ms / 13843 tokens [44325] 2.25.234.588 I slot print_timing: id 0 | task 0 | graphs reused = 2114 ubatch-size = 1024 [43573] 3.11.407.103 I slot print_timing: id 0 | task 0 | prompt eval time = 13709.73 ms / 11719 tokens ( 1.17 ms per token, 854.79 tokens per second) [43573] 3.11.407.106 I slot print_timing: id 0 | task 0 | eval time = 147916.76 ms / 2824 tokens ( 52.38 ms per token, 19.09 tokens per second) [43573] 3.11.407.107 I slot print_timing: id 0 | task 0 | total time = 161626.49 ms / 14543 tokens [43573] 3.11.407.111 I slot print_timing: id 0 | task 0 | graphs reused = 2812 Some takeaways: - a point to point network keeps the traffic only between these 2 nodes and not my switches - Use device=rpc0/rpc1 to restrict the worker CPU if you dont intend to use it.. default fit will skip the host cpu, but use the rpc cpu. - ubatch 768 was a sweet spot for this setup: Higher ubatch = Higher PP Lower TG , Lower ubatch = Lower PP Higher TG - split-mode tensor does not work in this setup, not solely because of network cap.. it just crawls to 1t/s Built with NCCL and RPC: diff .devops/cuda_rpc_nccl.Dockerfile .devops/cuda.Dockerfile 35c35 < apt-get install -y gcc-{GCC_VERSION} g++-{GCC_VERSION} build-essential cmake python3 python3-pip git libssl-dev libgomp1 libnccl2 libnccl-dev --- > apt-get install -y gcc-{GCC_VERSION} g++-{GCC_VERSION} build-essential cmake python3 python3-pip git libssl-dev libgomp1 44d43 < ENV LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu:{LD_LIBRARY_PATH} 49c48 < cmake -B build -DGGML_NATIVE=OFF -DGGML_CUDA=ON -DGGML_RPC=ON -DGGML_CUDA_NCCL=ON -DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON -DLLAMA_BUILD_TESTS=OFF {CMAKE_ARGS} -DCMAKE_EXE_LINKER_FLAGS=-Wl,--allow-shlib-undefined . && --- > cmake -B build -DGGML_NATIVE=OFF -DGGML_CUDA=ON -DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON -DLLAMA_BUILD_TESTS=OFF {CMAKE_ARGS} -DCMAKE_EXE_LINKER_FLAGS=-Wl,--allow-shlib-undefined . && 81c80 < && apt-get install -y libgomp1 curl ffmpeg libnccl2 --- > && apt-get install -y libgomp1 curl ffmpeg --- Playground: 2x 4060ti, 1x 3900x, 4x32GB Tequila: 1x 4060ti, 1x 9600x, 3x48GB # llama-server sees all devices as usable: 8.37.349.848 I srv load: /app/llama-server 8.37.349.850 I srv load: --rpc 8.37.349.850 I srv load: 10.44.0.2:50052 8.37.349.853 I srv load: --device 8.37.349.854 I srv load: CUDA0,CUDA1,RPC0 ... [60669] 0.03.803.724 I cmn common_param: device_info: [60669] 0.03.803.783 I cmn common_param: - CUDA0 : NVIDIA GeForce RTX 4060 Ti (15976 MiB, 15722 MiB free) [60669] 0.03.803.801 I cmn common_param: - CUDA1 : NVIDIA GeForce RTX 4060 Ti (15977 MiB, 15722 MiB free) [60669] 0.03.803.806 I cmn common_param: - CPU : AMD Ryzen 9 3900X 12-Core Processor (128219 MiB, 128219 MiB free) [60669] 0.03.804.960 I cmn common_param: - RPC0 : 10.44.0.2:50052 (15976 MiB, 15772 MiB free) [60669] 0.03.805.398 I cmn common_param: - RPC1 : 10.44.0.2:50052 (142083 MiB, 142083 MiB free) # llama-rpc-server (tequila) sees the full details for rpc0 vs rpc1 Starting RPC server v4.0.3 endpoint : 0.0.0.0:50052 local cache : /data/rpccache/rpc/ Devices: CUDA0: NVIDIA GeForce RTX 4060 Ti (15976 MiB, 15836 MiB free) CPU: AMD Ryzen 5 9600X 6-Core Processor (142083 MiB, 142083 MiB free) transport : TCP submitted by /u/Chuyito [link] [comments]

Source: r/LocalLLaMA | 2026-07-23

Related