Trying to setup two LLM’s to run on 2 gpus separately but simultaneously on one machine.

Okay so this probably sounds like kind of a dumb setup but hear me out. I have a 4070 I’ve been running gemma4 off fine but I recently slapped in a spare 1650 I’ve had laying around to run a second lighter weight model. I’m trying to set up Ollama in a way where I can run gemma4 on the 4070 and a lighter model like phi4-mini on the 1650. The issue is the power gap between the two is very significant that gemma4 would start nuclear war on the 1650, so I would need to assign each model to a GPU, or prevent the heavier model from loading on the 1650. At first I tried two services running locally on Linux but struggled to get the second service to bind to another port. Even then I’m unsure if I can get these to load the way I wanted to. Currently I’m trying to figure out docker to see if I can’t accomplish this that way, but am currently dealing with some dumb issues there. I wanted to see if anyone has done something similar or has any tips to make this setup work. The end goal is to have each LLM automatically run off their own GPU, so I can then set them up as two separate agents that can run simultaneously in Openclaw. submitted by /u/J-RodRS [link] [comments]

Source: r/ollama | 2026-08-02