Anyone else doing eGPUs (OCuLink)?

Upgraded to a 5070 Ti so I could run Qwen 3.8 27B, which works perfectly, but didn't want to let the old 4070 Ti go to waste. The cards would touch if I put them both in the PC and I knew the heat would be awful from my crypto mining days. I always was curious about eGPUs so I slapped together this hillbilly OCuLink enclosure and added a PCIe 4.0 16x OCuLink card to the PC. Anybody else doing this? Tips? Gotchas? Is this gonna end in tears? EDIT: OK this exceeded ALL expectations. Took almost no time at all to get back to 50 TPS at full context and q8_0 KV cache. I can't believe there is no loss especially at higher cache quant and max context. I was doing 112k and q4_0 on just the 5070 Ti. One oddity on latest server-cuda13, I had to reverse tensor-split. Instead of "N0,N1", I had to set "N1, N0". Noticed in nvtop it was going heavy on the 4070 Ti. I may try my own build next because: "W NCCL not compiled in; falling back to internal AllReduce. Recompile with -DGGML_CUDA_NCCL=ON for best multi-GPU performance." Here's what I'm using at the moment: # Global configurations applied to all models unless overridden [*] load-mode = none ctx-size = 131072 fit = on # Leave 450MB for the display driver fitt = 450 # IMPORTANT: Only use this fitt setting on a headless server. # This will tell llama_server to take 100% VRAM. # It normally reserves 1GB for system by default. # You will probably crash. #fitt = 0 fit-ctx = 131072 fa = on reasoning-preserve = true temp = 0.6 top-p = 0.95 top-k = 20 min-p = 0.0 repeat-penalty = 1.0 ngl = -1 np = 1 split-mode = tensor main-gpu = 0 tensor-split = 12,16 #ffn_cpu_odd #ot = blk.(1|3|5|7|9|11|13|15|17|19)ffn_up=CPU [Qwen3.8-27B-UD-IQ4_XS] model = /models/Qwen3.8-27B-UD-IQ4_XS.gguf ctx-size = 262144 #ctx-size = 112000 presence-penalty = 0.0 cache-type-k = q8_0 cache-type-v = q8_0 ngl = 999 fit = off t = 8 submitted by /u/krumpstead [link] [comments]

Source: r/LocalLLaMA | 2026-08-26

Related