Trying to optimize LLM inference latency — Ollama + OpenWebUI/AnythingLLM

Hi everyone, I'm currently working on optimizing an LLM setup at my company and I'm trying to identify where the actual performance bottleneck is. Our current setup is roughly: LLM: NVIDIA Nemotron model Inference: Ollama Interface/orchestration: OpenWebUI / AnythingLLM GPU: [GPU model + VRAM] RAM: [RAM] Current generation speed: 20 tokens/sec GPU utilization: 95% The main issue we're trying to solve is latency and overall response time. I've already gone through the configuration and checked the major settings on the Ollama/model side. From what I've measured so far, the model is actually utilizing the GPU heavily and generation itself seems reasonably consistent. What I'm trying to figure out now is whether the additional latency is coming from the layer around the model rather than the model itself. For example, I'm wondering about: Does OpenWebUI or AnythingLLM add significant latency through prompt construction, conversation history, RAG, middleware, or API calls? How can I accurately measure TTFT (time to first token) vs prompt processing time vs generation time? What metrics should I collect to determine whether the bottleneck is Ollama, the UI/orchestration layer, networking, or the model itself? Are there specific Ollama settings that have a major impact on latency that I should benchmark? How much can large context windows / conversation history affect inference performance? If OpenWebUI/AnythingLLM is sending a much larger prompt to Ollama than the user-visible message suggests, what is the best way to verify that? What would you recommend for systematically benchmarking the same model directly through Ollama vs OpenWebUI vs AnythingLLM? My goal isn't just to make the UI feel faster. I want to identify exactly where the latency is being introduced and then optimize that component. I've already checked the obvious configuration issues, so I'm particularly interested in practical benchmarking/debugging methods or experiences from people who have optimized Ollama + OpenWebUI/AnythingLLM deployments in production. If you need more information about the setup, I can provide the exact GPU/model/configuration and benchmark results. Thanks! submitted by /u/daddyMaterialBolte [link] [comments]

Source: r/ollama | 2026-08-23

Related