Built a token-aware gateway/load balancer for local LLM stacks — because nginx has no idea what a token costs

If you're running Ollama, llama.cpp, or vLLM behind nginx or HAProxy for more than a single user, you've probably hit this: nginx treats a 10-token prompt and a 10k-token prompt as identical "one request" — it has zero concept of how much GPU memory a request will actually need. Under load, that means the KV-cache fills unevenly, latency spikes, and you can end up with OOM instead of graceful degradation. I built TokenFlow Gateway to fix that at the proxy layer, specifically for local/self-hosted stacks: - Pre-flight token estimation (js-tiktoken) before dispatch — request weight = prompt tokens + max_tokens - KV-aware load balancing for vLLM — polls its Prometheus metrics (vllm:gpu_cache_usage_perc) and routes heavy prompts to whichever backend actually has room - Works with Ollama too — since Ollama doesn't expose KV-cache metrics, it falls back to health checks plus its own in-flight token accounting (reserves each request's estimated weight until it completes) - Admission queue instead of crashes — Redis-backed, priority-aware, configurable timeout when nothing fits - Exact + semantic caching (hash + pgvector) so repeat and near-duplicate prompts never touch the GPU twice - Per-API-key token-based rate limits (TPM/RPM), OpenAI-style, for your own hardware It's OpenAI-API-compatible on both sides, so it drops in front of Ollama, llama.cpp server, vLLM, or TGI without any client changes. No GPU needed to try it: docker compose up --build spins up the gateway, Redis, pgvector, and two mock backends that mimic the OpenAI API and vLLM-style metrics, plus a smoke script that fires a burst of concurrent long-context requests so you can watch it queue and balance instead of falling over. Stack: TypeScript / Fastify / Redis / Postgres+pgvector. MIT licensed. Repo: https://github.com/mosafariuk/TokenFlow-Gateway Would love feedback from anyone running a multi-model or multi-user local setup — what does your current ingress/routing look like? submitted by /u/Electrical_Emu_5854 [link] [comments]

Source: r/ollama | 2026-08-17