Local Ai
Tried running LLMs locally to save API costs… ended up waiting 13 minutes for ONE response 🤡
A Reddit post in r/ollama describes a user's experience attempting to run LLMs locally via Ollama to avoid cloud API costs, only to encounter severely degraded performance — waiting 13 minutes for ...
A Reddit post in r/ollama describes a user's experience attempting to run LLMs locally via Ollama to avoid cloud API costs, only to encounter severely degraded performance — waiting 13 minutes for a single response. This outcome is consistent with a well-documented pitfall: extremely slow generation (under 5 tokens/second) typically indicates CPU-only inference when GPU acceleration was expected , and tools like Ollama process requests sequentially, meaning latency can spike dramatically — from 2 seconds to over 45 seconds — even under modest load . The post highlights the importance of matching model size to available hardware, as RAM matters more than most developers expect — models load into RAM before GPU processing, and running a 7B model with an 8k context requires 8–10 GB of RAM .
Related
- Recommended Model for a 4060ti 8gb and 16gb ram
- Mac mini M4 48GB
- How do I know if an AI model could work locally on my computer?
- Use the Same Model Across Ollama, LM Studio, Jan, and your Favorite Local AI Apps
- Is the ASUS ROG Flow Z13 with 128GB of Unified Memory (AMD Strix Halo) a good option to run large LLMs (70B+)?
Source: local-ai