Running an AI SaaS against local Ollama models — what actually breaks when you swap the provider

I maintain a starter kit where the LLM provider is a runtime-swappable DB row, not a hardcoded constant — OpenAI, Anthropic, or a local Ollama instance. Mostly so I can demo without burning API credits. Sharing the friction points, since "just point it at localhost" understates it. How a local model gets wired in (no deploy, it's an admin action): POST /api/providers { "name": "Local Llama", "provider": "ollama", "baseUrl": "http://localhost:11434", "isActive": true } Then the chat code is unchanged — the model name is resolved from the active provider's default, so the request can even omit it. That's the whole point of provider-as-data. What actually breaks, in order of how much time it cost: Params that don't exist. OpenAI chat params (frequency_penalty, presence_penalty, response_format, tool choice) aren't all supported by every Ollama model. LiteLLM's drop_params=True silently drops unsupported ones instead of 400-ing: litellm.drop_params = True # Drop unsupported params for providers like Ollama This is the right default for a multi-provider app but it means your carefully tuned temperature might just... not apply on the local model. Verify per-model, don't assume. Streaming frames differ. OpenAI, Anthropic and Ollama don't frame SSE deltas the same way. Ollama in particular emits usage-only and role-only chunks. If your client does "for chunk: append chunk.content" without a guard, you get nulls or a truncated answer. We normalize every chunk into one internal schema before it hits the client. Embeddings for RAG. The default embedding model is OpenAI's text-embedding-ada-002 (1536-dim, FAISS IndexFlatL2). If you want fully-local RAG you need a local embedding model too, and the vector dimension of your FAISS index has to match the model you embedded with — swap models and your existing index is silently wrong. Re-ingest, don't just repoint. Latency and the "200 OK then error" trap. A cold Ollama model load can take many seconds. Because streaming responses send 200 immediately, a timeout during load can't become a 500 — it has to surface as an SSE error event mid-stream. Easy to miss locally because on a warm API it never happens. None of this is Ollama's fault — it's the cost of pretending every provider speaks one protocol. The abstraction is worth it for dev/demo, but I would NOT run a paid product's only path through it without per-model testing. Curious what others do for #3 — for local-first RAG are you using nomic-embed / bge / mxbai and re-embedding your corpus per model, or keeping a separate OpenAI-embedding index for anything that needs to be searchable in prod? submitted by /u/Big_Burger_2904 [link] [comments]

Source: r/ollama | 2026-09-25

Related