Echo Dot 2 can run 28M LLM at decent speed

Code and instructions available here: https://github.com/albertoZurini/echo-dot-2-playground Hello there! After a few days of experimenting I was able to get a completely local voice pipeline running on an Amazon Echo Dot 2. The interesting part for this community is that the device can run llama.cpp alongside offline speech recognition even with its very limited hardware. The Echo Dot 2 has an ARMv7 processor and 512 MB of RAM. I compiled llama.cpp for armeabi-v7a and run the model through a persistent llama-server process. Keeping the server alive is important because otherwise the model gets loaded from scratch for every request. For the smaller experiment I used a 28M parameter model, the same general model family used in some ESP32 voice projects. On the Echo Dot it reaches roughly 7 tokens/s during prompt processing and about 4 tokens/s during generation. That is obviously not fast, but it is enough for short structured commands such as turning a light on or changing the volume. I also tested MobileLLM-143M-Q4_K_M.gguf. It works, but a complete prefill and generation took around 20 seconds for a simple command, which makes it feel too slow for an interactive voice assistant on this hardware. The biggest improvement came from changing the inference architecture. Instead of starting llama-cli for every utterance, the assistant starts llama-server once and leaves it resident in the background. The request prompt keeps all invariant instructions and the output format at the beginning, with the user's text at the very end. The request uses cache_prompt=true, so llama.cpp can reuse the KV cache for the stable prefix and only process the changing user text. In my tests this reduced warm query latency from roughly 17 seconds with a cold llama-cli invocation to around 2.3 seconds with the resident server and cached prompt. The rest of the pipeline uses Sherpa-ONNX for offline streaming speech-to-text. The wake word service sends a local event to the assistant, the assistant records speech, runs STT and then sends the transcript to the local llama.cpp server. The model is constrained to emit simple JSON actions instead of a conversational response, which keeps generation short and makes the result useful despite the low token rate. This is not a general-purpose chatbot. The model is too small and the hardware is too slow for that. It is more like a tiny local intent parser that can turn speech into device actions without sending audio or text to the cloud. I was able to trigger simple tools by voice and keep the full interaction on the Echo Dot. The code and build notes are here: https://github.com/albertoZurini/echo-dot-2-playground I would be interested to hear what other small GGUF models or llama.cpp settings people would try on this kind of old ARM hardware. The main constraints are RAM, model loading time and keeping the prompt small enough that the cached prefix actually helps. submitted by /u/alberto_zurini [link] [comments]

Source: r/LocalLLaMA | 2026-08-07

Related