Six days building a local assistant on a 16 GB Mac. The numbers that changed my design.

I spent the first week of September building a personal assistant over Gemma 4 in Ollama on a 16 GB M1 Pro: Node backend, React web UI, Expo phone client, two-model routing, cross-session memory, RAG over my notes, web search through tool calling, and a coding agent behind a command policy. Most of what I assumed going in turned out to be wrong, so here are the measurements that actually changed the design. The two models do not fit together, whatever "ollama ps" says. ollama ps showed 3.2 GB for gemma4:latest and 8.1 GB for gemma4:12b, so I planned on both resident. Ollama's scheduler predicts 10.6 GiB for the e4b build (about 6 GB of embedding tables sit in host memory and never show in ps) and evicts the other model every time. Each swap is a 4 to 7 s cold load. Switching to gemma4:e4b-it-qat dropped the footprint to 6.4 GiB and went from 27 to 33 tok/s, but still not co-resident with the 12B. The router became "sticky": fast by default, deep only on strong signals, and once a conversation is on deep it stays. At most one swap per conversation. Where the date sits in the prompt cost me 1.5 seconds per turn. Ollama reuses the KV cache for the longest shared prefix. With the current date at the top of the system prompt, every follow-up re-evaluated the whole conversation: 1.8, 1.9, 2.0 s to first token. Moving everything that changes per message (date, retrieved chunks, recalled memory) into a marked block at the top of the user turn, and keeping the system prompt stable, took follow-ups to 0.5 and 0.4 s. Nothing else changed. The system prompt suppresses tool calling, and nudge position matters more than wording. gemma4-fast with web_search offered, 9 prompts (5 needing a lookup), seed 7: - bare prompt, no system message: 8/9 correct - my full system prompt (about 700 tokens of "say when you don't know"): 5/9, missed 4 lookups - same, with a one-line "this needs current information, search first" placed in the context block BEFORE the question: 5/10, worse than no nudge - same nudge placed AFTER the question, last thing before generation: 9/9, zero needless searches Gemma has no system role, so Ollama folds the system prompt into the first user turn, and a small model reads "say when you don't know" as permission not to look. The failure mode wasn't a refusal either: with the nudge in the wrong place it answered "32°C and partly cloudy [weather.com]" without ever calling the tool. I now have a separate rule that it may only cite a domain that appeared in an actual tool result. Memory: injecting everything cost 2,300 tokens a turn. The idle-time extractor had accumulated 109 facts about me. Injecting all of them was half the fixed cost of every prompt. Now preferences and pinned facts go in the system prompt, and plain facts are embedded with nomic-embed-text and the nearest 10 within L2 1.0 are recalled per message into the user turn. Ten recalled facts cost about 150 tokens. Rewordings of the same fact sit 0.12 to 0.30 apart, different facts about the same subject 0.53 and up, so anything within 0.35 replaces the old wording instead of piling up. Also learned the hard way: the extractor once stored "the user wants replies of exactly three lowercase words describing the sea" (a test prompt) as a standing preference. Every answer afterwards was terse and lowercase until I found it. Small models over-collect; every fact needs to be one click from deletion. Coding agent: default to confirm, not safe. Same tool loop as web search, ten tools (read, search, edit_file, run_command, git inspection). edit_file with exact old text works far more reliably on a small model than unified diffs. Every command is split on shell operators and each piece is classified safe / confirm / deny; anything unrecognised is confirm. The UI tags test commands "passed" or "failed" from the real exit code, not from what the model says, because the model will say tests passed when it never ran them. I wrote up the full thing (architecture, all the numbers, the order I'd build it in again) as a single HTML doc. Happy to share it in the comments if there's interest. Stack: Node/TypeScript/Fastify, SQLite + sqlite-vec, Vite + React, Expo. Ollama for the models, Whisper in-process (transformers.js) for dictation so audio never leaves the machine, Tailscale between phone and Mac. Question for people who've done this on 16 GB: has anyone found a fast/deep pair that actually stays co-resident with something 9B or larger as the deep model? qwen3.5:4b at about 2.5 GB looks like it should, but I haven't measured it yet. Some of the screenshots of my local chat assistant on both Web and Mobile interface: https://preview.redd.it/6xz6kruzb7qh1.png?width=2880&format=png&auto=webp&s=1491727e3c307906433c7f99eb0679d2a2932cd3 https://preview.redd.it/hoib8ruzb7qh1.png?width=1080&format=png&auto=webp&s=237c8596bd584ccbe04c1319db3b0686d839b6f0 https://preview.redd.it/vwm8vsuzb7qh1.png?width=2880&format=png&auto=webp&s=d6557d999a2e48cc0e395dc22bec359fc6d52048 https://preview.redd.it/t3wgwruzb7qh1.png?width=1080&format=png&auto=webp&s=9a1b463b36315c635da6004e079e4ed21670855c https://preview.redd.it/4v75jquzb7qh1.png?width=1080&format=png&auto=webp&s=1dc9090a72bc722e2c51b2fe6087fd3e771c1579 https://preview.redd.it/wym35suzb7qh1.png?width=1080&format=png&auto=webp&s=0b3718d019e158a58df96cd66a6684a44974a64c https://preview.redd.it/mif7tquzb7qh1.png?width=2880&format=png&auto=webp&s=f5e6fc235ea4257b6dd5ea3795f6514e569926db https://preview.redd.it/l9ul8quzb7qh1.png?width=2880&format=png&auto=webp&s=14caf42477f506a6f5a9556e667fe854feaf4396 submitted by /u/Namish_Mudgal [link] [comments]

Source: r/ollama | 2026-09-17

Related