New challenge: consolidated 2 cards in a single system but one card hogs all the models even if I run 2 different instances of Ollama on 2 different ports

In my attempt to combat powerty; as electricity cost a ton these days and it is hot as hell, I consolidated 2 machines into one, so my 2 crappy machines now are one, and that means a bit less power and some cash in my pocket and a happy kid on the block that can play fortnite I guess :) Anyway; I have one AMD card running on a Gen 1 X4 PCIe bus, and the main Nvidia card is running on the Gen3 X16 Pcie bus slot; both are happy, drivers are happy. I set up 2 instances of Ollama: one on port 11434 and one on 11435; this is running on linux as OS. The first instance runs with CUDA_VISIBLE_DEVICES=0; which if I am getting this right, is telling Ollama to run that model on my main CUDA card, which is my slot 0 Nvidia card. The second instance of Ollama has ROCR_VISIBLE_DEVICES=0; which again if I got this right, is telling Ollama to run the model on the first ROCm device, which is the AMD card on slot 1. But it does not work... No matter what, when I load a model on either instance on port 11434 or port 11435, the AMD card load the model. Been going back and forth for a bit, and can't get this solved. Isn't this something that should work out of the box once you force the hardware in the service file? EDIT --------------------------------------------------------------------- Solved: I had no idea that by default, Vulkan just enable itself and grab both cards. So they were both under Vulkan. Once I turned off explicitly Vulkan when I use the Nvidia card, everything seems to work fine, and the Nvidia card is loading correctly a model while the AMD card is loading another model, on their respective instances on their respective ports submitted by /u/fttklr [link] [comments]

Source: r/ollama | 2026-10-08

Related