Knowledge catalogue
You guys remember sesame AI? With their really good AI voice model? Obviously ChatGPT has their voice model that’s also really good. Is there any smaller local variant that runs on like consumer grade
specs hardware: M4 Max 128GB Studio inference engine: llama.cpp (qwen4exp branch) judge: claude-opus-4-6 AtomicChat/Qwen3.8-Flash-Next-GGUF Qwen3.8-Flash-Next is a great model I benched in my previous
I put together a technical guide for installing AutoGPT from source, with a focus on using local models instead of OpenAI. It includes Docker Compose files, environment variables, and how to point the
This has been a big 2 weeks for local models with 3.8 27B and Next coming out, a lot of 'frontier labs are done' comments which, in general, could certainly be true. Could be trillions in economic val
Hey all, I stumbled onto something funny and figured I'd share before it gets buried. I'd been running the Ornith 1.5 35B A3B GGUF (Q4_K_M) for a while, then recently a new revision showed up in my HF
I absolutely love Ollama for the majority of my inference needs. Starting to roll out Rehex to friends, and I've gotten requests to support OpenCode Go and Codex. Frankly, I'm really excited to add th
In mid-August, Ramp published spending data collected from 70,000 U.S. companies: Fable 5 ,the most powerful and expensive model in Anthropic’s lineup, accounts for just 11% of what those businesses s
I have a 2 DGX Spark setup recently and I have been happily running Deepseek V4 Flash 0731. Since the release of GLM5.3 Flash and Qwen 3.8 Flash Next this week, a lot of folks are still waiting to see
Hardware: rtx 5080 16 gb vram; 64 gb ram ddr5 6000hz; ssd with unlimited memory; ryzen 7 9800 x3d. OS: Windows 11 Software: I’d prefer llama.cpp, but it’s not a strict requirement; I’ll use whatever y
People who moved from older version to this new one, how much improvements do you see on inference? Ex: llama.cpp performance? This new version comes with Linux Kernel 7.0. Hope AMD cards gonna enjoy
Folks! We're just 50 PRs away from more faster inference. Hopefully by end of year. Experts!, please chip in there. List of Open/Ongoing PRs(and also Discussions) related to CPU/RAM/Disk/Hybrid: [Disc
I kept running into the same thing. Change a system prompt, get a different answer, and I couldn't tell if that was my change or just the sampler. The loop made it worse. Edit the Modelfile in another
Just FYI. They removed it. Free models that can be used with Ollama Cloud now: gemma4:31b gpt-oss:120b gpt-oss:20b nemotron-3-nano:30b nemotron-3-super nemotron-3-ultra submitted by /u/IndividualPlus2
TL;DR: Every public low-bit GGUF of this model is secretly ~4.70 bpw. Shim the rows to 256 and it becomes a real 3.07 bpw / 11.77 GiB file that runs 262K context on 16GB. Needs patched llama.cpp — not
Hey! I forked NInfer (a from-scratch C++20/CUDA inference engine for Qwen models) and added two things: tensor-parallel across two GPUs, and YaRN ×4 rope scaling. Together they let Qwen3.8-27B NVFP4 r
happened simultaneously across the board. OpenCode, OpenRouter, and Ollama all ruined DeepSeek at the exact same moment. They literally turned a brilliant model into a complete idiot overnight. I abso
I wanted to share my successful setup for running a Qwen 3.8 27B model with a massive context window on a consumer 16GB GPU (RTX 4070 Ti SUPER). The goal was to fit everything into VRAM without sacrif
I ran a few benchmarks on my MacBook Pro M5 Max with oMLX: Running Qwen3.8-27B with thinking off has a big effect on output quality. Running it with thinking on xhigh burns 5.5x the tokens and runs 6x
Below are the benchmark results for running Qwen3.8-Flash-Next on Strix Halo using the Vulkan backend of llama.cpp, combined with MTP model. Hardware Item Details CPU AMD Ryzen AI MAX+ 395 (16C/32T) G
~20–30GB smaller than Unsloth/AesSedai Q4 at similar PPL. After several days of testing I released a set of mainline-compatible imatrix quants for Qwen3.8-Flash-Next. Goal: same quality band as the po
this model is really something incredible, saved us like 600 dollars. my fiance is always breaking her electronics and then getting me to fix them. The other day she brought me her phone and it was st
I saw someone here posted about getting a Tenstorrent QuietBox 2, and I wanted to look into it the hardware. It's very difficult to find any benchmarks, but I managed to find some from an employee. Th
Announcement: https://www.tbench.ai/news/terminal-bench-4-0 Leaderboard: https://www.tbench.ai/ Imo the best aspect in their announcement is their focus on rapidly iterating on TerminalBench to keep t
Hi r/ollama! I am the solo dev of PowerChat, an AI chat client that provides a mobile chat interface for iOS (iPhones for now, iPad and Mac planned) to connect to self-hosted models using Ollama, or O
The official benchmark card for Ling-3.0-flash-Fin is a useful reminder that the unit being tested is rarely just “the model.” The release says most runs used temperature 1, top_p 0.95 and the highest
I was running Ollama with Maxwell 1500, using a 65,536-token context window, through Zoo Code inside VS Code. I asked the model to analyze a folder containing some pretty simple code. It spent around
hi i make tool for this called llmog it's purpose to make llms free to - auto annotation datasets - reclassification existing yolo datasets running totally local using llama cpp or vllm or use externa
This base model is only 43m parameters trained on 3m arXiv abstracts. We plan to continue pre-training and post training. If you create fine-tuning datasets or if you know of any datasets that can hel
Quote: ... developing a new A.I. model intended to be as powerful as Anthropic’s cutting-edge models. ... In July, while developing Watermelon, Meta paused and later resumed a stage of A.I. developmen
We are considering to buy computer, server to run decent model on premises. I would like to run big open source model with more that 70b parameters roughly. I read the people used to run on Apple Stud
Hey guys, FastVideo team here. We saw how important speed and quality is for everyone. And we've been working to create our own step distill checkpoints and LORAs for MINIMAX h3. Here's is our v1 rele
Hey, An LLM writing into durable memory is an unreliable author. It invents, mislabels, restates, and contradicts. Once one of those mistakes is saved, the agent can carry it into every conversation t
Model: unsloth/GLM-5.3-Flash-GGUF UD-Q2_K_XL (4 shards, 101 GiB) + mmproj-BF16. Reference: llama.cpp PR #27754 (glm5next), CUDA 12.8, SM 120. Throughput Both engines back to back in one session, flash
Happy Friday! Ever had issues getting the anatomy right on your t2v generation? Just add a slider with your H3 Prompt! What have you all been building on your local AI studios? 384x448, int8, 20 steps
As in actual physical heat. How are people coping with the heat that running a decent inference rig pumps out? I've got dual 5060 Ti GPUs and a relatively modest CPU (Intel 14 Core Ultra 5 245KF Deskt
Has anyone been able to get refund from Ollama cloud? What is the process to get it. There are cheaper alternative now and I was stupid to pay for a full year cos it seemed lucrative then. submitted b
So me and various other people have implemented their own Sparse Attention nodes and you can see many people argue about what % you should actually run these nodes on to maintain prompt adherence etc.
I dont know about others, but Nvidia seem to close the lid on older GPUs since they want to push their new technology. Llama and team has been the to go places for older GPUs like V100s. Knowing how N
TL;DR: k-quants need tensor rows divisible by 256. When they aren't, llama-quantize quietly swaps in a ~4.5 bpw type and the file keeps its low-bit name. I audited 443 quants across 25 repos; 64 are a
I built a benchmark called EchoNet. An agent gets a factual question, then searches a syntheic web I made before answering. Some of that web is seeded with misinformation: one fake page, a fake page r
I fine-tuned and published SaphraFgt 1.0, an 8B coding model focused on HTML, CSS, JavaScript, frontend development, UI/UX, code completion and debugging. Ollama: [https://ollama.com/SaphraFgt/saphraf
Smaller models still struggle with browser tasks that require open-ended research or deep reasoning. But many practical browser-automation workflows are much more procedural: extracting structured dat
I've been running Qwen3-0.6B on the M5Stack LLM-8850 card (Axera AX8850 NPU, 24 TOPS, 8GB LPDDR4x) hosted by a Raspberry Pi 5 — as a llama.cpp backend. The problem: the vendor stack requires convertin
However, I keep getting a 404 error when trying to access qwen3.8-flash-next:125b-mlx and ornith-1.5:397b via the API, even though they are listed in the model list. Does anyone know why this is happe
The initial reason for this was the Comfy H3 Sync & Sound Community Challenge: Comfy H3 Sync Sound Community Challenge! - by Allyson Toy I made a short rap track in Suno, then used Hermes Agent to bui
I read this somewhere and wasn't sure if it's still an issue or if it only affects external models versus all models used via the Claude Code CLI. 'This latest Ollama update finally fixes that issue w
Using https://huggingface.co/RadixArk/Qwen3.8-Flash-Next-NVFP4 and https://old.reddit.com/r/BlackwellPerformance/comments/1w04xb7/qwen38_flashnext_on_1x_rtx_pro_6000_171_ts_c1_428/ As usual, all the d
Built the latest and I'm getting as much as 220 tokens per second and averaging in the 170s, I can't get over it. If anyone on here is on that project, fuckkkin' chapeau man, really incredible job. I
Proof is in the method honestly. Closed model labs need to constantly reinvent the wheel to keep lead. Open source has a bunch of independent labs practically working somewhat together. Eventually whe
A few days ago I posted asking what people considered the best local model for an 8 GB VRAM GPU. At the time, my personal sweet spot was Qwen3.6-35B-A3B, for agentic coding with Pi.dev. Well… Thanks t
I wanted to see whether NInfer could run Qwen3.8-27B on an RTX PRO 4000 Blackwell 24GB. Hardware/software: NVIDIA RTX PRO 4000 Blackwell, 24GB 145W power limit CUDA 13.3 NVIDIA driver 610.x Ubuntu LXC
EDIT: Though the issue with q8 kv cache seems to arise from when and how often we run the quantize step, not that kv quantizing can't ever work - see comments for various proposed solutions --- One of
TL;DR: llama.cpp with --load-mode mmap used 21-32 GB of RAM, with ik_llama.cpp using 106-108 GB. -sm tensor killed my prefill, changing to -sm layer went from 36 tps to 135 tps. After that, -ubatch 20
I've got Qwen3.8-Flash-next running on RTX 3090, Ryzen 9 3950X, a PCIe 3.0 motherboard, and 64GB DDR RAM from 2020. IQ4_XS weights, full kvarn5 context, vision on GPU, experts in host RAM, n-grams on
We're releasing Qwen3.8-27B quantized with our newest methods, GSQ + RCO. Higher-quality models, same file size, now with the search and the quantizer both learned. What's inside: GSQ (Gumbel-Softmax
Their last version 7.14 was released just a month ago. llama.cpp PR(waiting for approval) for Version 10.0 https://github.com/ggml-org/llama.cpp/pull/27803 Hope this version comes with more boost & im
Row-Bot v4.9.0 is available. - Meet Buddy: a native, always-on-top desktop overlay for Windows and macOS. - Drag Buddy from the sidebar and place it over any app. - Chat, track progress, read replies,
Been building AgentSwarms for a while. Agent platform with RAG, multi-agent workflows and a BI layer over your own data. Posting here because the whole thing runs without a single hosted API if you wa
TL;DR: Self-learning needs governance, observability, and auditability. Sounds great, it's not great unless you have serious and appropriate protections. Let me lead this off with what we are most pro
Post rédigé avec l'aide de GLM5.3 flash Contexte : formateur professionnel d'adultes en France, j'utilise des agents LLM au quotidien et le français qu'ils produisent est mon critère n°1, avant les be