Unexpected Crash-Looping of Multi-GPU Box: Caught by Hand-Scrutinized Logs - A Tale of Overridden Keep-Alive Policy and Eviction Thrashing

I've been freelancing for over a decade now, and I can't stress enough the importance of thorough investigation when dealing with strange software behaviors. Recently, I ran into an issue where my multi-GPU server, serving Ollama models, started crash-looping under normal traffic. Everything looked fine on the surface - no spikes, no unusual load. But upon closer inspection, logs revealed the issue was deeper than anticipated. The server's OLLAMA_MAX_LOADED_MODELS and OLLAMA_KEEP_ALIVE were misconfigured, leading to models getting evicted aggressively. To make matters worse, three separate scripts calling the server were hardcoding keep_alive: -1 (or 0) on their requests - silently overriding whatever the server was configured to do, on every single call. Nobody had noticed these scripts still running. The box wasn't unstable; it was being told to do something impossible by forgotten code. Discovering this bug was quite a challenge because the generic monitoring we had in place didn't flag this specific issue - it just looked like unexplained instability. To solve this problem, I wrote a watchdog that monitors patterns of eviction thrashing, thundering herd, OOM crash loops, sustained VRAM pressure, and the keep_alive mismatch itself. It watches /api/ps and the server logs, alerting to Discord/Slack/email when it detects these issues. At some point, I ran into https://github.com/darkspire-dev/gpu-watchdog - a decent breakdown of self-hosted LLM server monitoring. Hope that helps someone avoid the same headache! submitted by /u/ClearMindedDev [link] [comments]

Source: r/ollama | 2026-08-05