Knowledge catalogue
The Artist: Qwen3.8-27B-UD-Q3K_XL, q8_0 caches, xhigh, temp 1.0, image-min-tokens 1024, froggeric template) I was screwing around with different Qwen3.8-27B quants and thought of this very simplistic
We're releasing a fully quantized NVFP4 version of Qwen3.8-27B. The checkpoint was trained using quantization-aware distillation (QAD) with QUASAR, our new QAT algorithm. We used the original BF16 mod
Hi all, I'm looking for the best model for a hobby project and trying to make sense of the various data I came across. I know benchmarks do not often translate to the real world, especially to your pa
So I've been going down a rabbit hole trying to run local LLMs without dropping 800+ on a Mac or GPU. RAM prices are insane right now and I realized that used phones have decent unified memory for dir
Hi everyone. I am currently using gemma 4 model and used qwen 3:8b model. My goal is to use claude code to make projects, AI Tools Please help me find out what is more useful. Suppose i have to make a
Hugging Face is exploring sale of the business valued at around $13 billion dollars. Actually I don't think we have any other repo source. Which has the mix of model weights, datasets and Spaces. Kagg
I think that mall models between 10B and 40B are the real gamechanger. These models will be the ones that will make the AI buble pop and big companies like Anthropic and OpenAi go to hell. We are curr
I ended up fine tuning a mistral 7b to outperform our costly foundational model and saved $300k. I previously thought that fine tuning was pointless (it's definitely not) and that all these problems c
https://preview.redd.it/hpnr51vgcnlh1.png?width=1089&format=png&auto=webp&s=e960ca848fb2fa1d5dff0868e099ae682bac8310 This is local, uncencored ollama submitted by /u/Producer_John-Ian [link] [comments
Hi everyone, it's been a while since I posted so here's an update on what the Lemonade community has been up to this summer. Our overall mission is to enable local AI builders with everything they nee
I was bored and handwrote a tiny 100-line bash script to let an agent search for and read articles from an offline wikipedia archive during a regular chat. It's not particularly useful, but it's defin
Megathread for discussing the release of GLM-5.3-Flash. Quants Fine-Tunes & Abliterations Chat Templates Inference Server Support & Configuration Experiences, Benchmarks & Model Comparisons We'll try
Megathread for discussing the (impending) release of Qwen 3.8 Flash Next. Quants Fine-Tunes & Abliterations Chat Templates Inference Server Support & Configuration Experiences, Benchmarks & Model Comp
Hi, So I’m running Qwen on M4 Pro (ollama) and it drives me nuts at times. For example it’s trying to use grep with flags which don’t work on Mac. Or it’s writing some python code and failing and then
So here's the thing, almost everyone use NVIDIA to run their LLMs, we also do the same, a lot of people we've met use like RTX PRO 6000 or even H100, B300 It seems like everyone eyes is looking at NVI
In the past, I have use llama.cpp, but I read that the exl3 quantization format should give better precision, so I have tried exllamav3/tabbyAPI. It was able to write the shown simple HTML game withou
Yet another vLLM fork thread here, but this time its for older INT8-centric hardware. This is a complete INT8 serving stack for Qwen3.8 27B based on vLLM, AITER, and a 27B GPTQ INT8 quant w/ DFlash2.
https://preview.redd.it/i8rjx0ar5mlh1.png?width=2160&format=png&auto=webp&s=c2588bc7b2519ea71b176ca73faf566dfc585496 I wanted to see whether a heavily quantized 27B model running entirely on an older
I am aware that I may not be able to run some higher end models, but I’m starting my journey into Ollama to learn capabilities and limits of local hosting. What could I expect to run on a Lenovo Think
Hello, I've been self-hosting LLMs on various budget hardware for a while (6x RTX 3060 12 GB, Intel Arc Pro B60 24 GB, RX 9070 XT, etc). Over the last few months, I wrote about it in 4 articles: Gener
Benchmarked qwen3.8 xhigh, medium and muse glimmer. Xhigh effort mode with qwen3.8 took almost 30hrs. (And still failed on 16 cases because of the 32K output token limit) Medium effort mode and muse g
we made vision mlx quants of qwen3.8 27b (9 builds from 8bit at 29.5 GB down to 3.23bpw DWQ at 11.8 GB) and compared them against other community vision mlx quants from hf (we only compared vision bui
ollama ps settles this in one line, and almost nobody runs it before changing their quant. The PROCESSOR column reports exactly one of three things: 100% GPU - every layer on the GPU 100% CPU - the GP
I ran 11 uncensored variants of Gemma 4 12B that I grabbed from huggingface, sorting by downloads. 10 full abliterations plus 2 LoRA adapters which were requested to be added in the comparison, agains
With hopes of a Qwen3.8-35B-A3B release now mostly dashed, many people including myself are looking at fine-tunes and other variants of Qwen3.6-35B-A3B to run on VRAM-limited hardware. I decided to tr
.. didn't llama.cpp aka ggml get acquired by Huggingface not too long ago? How would this sale affect llama.cpp and ggml? What are possible risks, and did ggerganov share his opinions or potential nex
'A 12-core GPU, also with two more cores than before, now includes Neural Accelerators in each core for the first time on Mac mini, resulting in up to 4x faster AI performance and 2x faster graphics t
TL:DR: power management was doing auto-suspend and dumping model weights into RAM. Models weights 18gb+ RAM 16gb = no Bueno immediate crash. amdgpu.runpm=0 in your grub disables runtime power manageme
One of the things I like most about Ollama is how low it makes the barrier to experimenting with local models. The downside is that once you move away from high-end hardware, finding the right informa
An RLM agent microharness that 'persistently' thinks continuously. What could go wrong :) Introducing Headlong, an open source microharness for persistent agents: self-guided agents that think continu
We are seeing every so often people mentioning using Kiwix or offline Wikipedia for training. It is not something Kiwix is actively involved in (as in, people found us rather than the other way around
Hello everyone. I hope is all well with you all. I been around this sub for a few months and been quietly reading and I have seen how many of you are spending $10k on the Mac M5 and I am assuming you
Granite-4.2-30B is the flagship reasoning model in the Granite 4.2 family. It delivers the strongest performance across reasoning-intensive tasks by leveraging built-in <think>...</think> chain-of-tho
Building a local AI agent hierarchy on a budget — RAG, quantization, and testing whether fact-grounding pushes small models toward real creativity I don't have a CS background, a GPU, or much of a bud
Hi, since ollama provide multiple cloud models i was wondering is there an app that could be used for chat like claude already has but you can switch models? Or if that is complicated to build at leas
I made a Discord for people who are genuinely into AI and want a decent place to talk about it. It’s still new, but the idea is to build a large community without arbitrary bans, hidden moderation dec
Krea 2 Turbo — 4-Step Distillation LoRA (work in progress) A LoRA for Krea 2 Turbo that reduces the minimum usable step count from 8 to 4. This is an update release, following up from my previous post
Hi, I’m having fun using “huihui_ai/Qwen3.6-abliterated:35b-Claude-4.7” for pen testing. It’s uncensored so it does whatever I want. I’m on M4 Pro 14 core 64GB. Just wondering if there are some models
Hey everyone, I'm trying to set up a local LLM on my machine. My goal is to use it with Qwen Code or OpenCode to help me out with small to medium sized tasks, but nothing too big. I tested many differ
I’ve been going back and forth on this for a week and I can’t settle it, so I’m hoping someone here has hands-on numbers. The two configs (German prices, dealer quote, incl. VAT): Config Price Mac Stu
At $10k, you could get - 6.2B tokens with Qwen 3.8 Max (Qwen Pro plan) - 5.7B tokens with DeepSeek V4 Pro OpenRouter - 100B tokens with DeepSeek V4 Flash OpenRouter As a firm believer of local inferen
For some reason I can only use Ollama through home wi-fi, everytime I try to use it while using my phone as a modem I get an error. Vpn doesn't help. Edit: problem fixed, works perfectly if VPN is in
We have been working on some performance optimisations for Qwen3.8 and other models. The main new feature that we introduced is adaptive speculation for Llama.cpp What is it? MTP and DFlash work well
Pretty simple, I tried almost anything I could find with Q4K_M and it refused any harmful test prompt I could find, does anyone know a decent versione/how fo make It work? submitted by /u/circumcised_
Portable rig for Qwen3.8-27B-BF16 200K+ token prompts. My work Panasonic Toughbook + the T1 + power brick + headphones all fit in my lunchbox. Need the BF16 for huge context highly sensitive document
good morning, has anyone already used qwen 3.8 with the tesla v100 or sxm2 or pcie? any information is useful to me context token/s thanks for the help. submitted by /u/Full-Chicken2110 [link] [commen
My best bet is the embedding-offloaded linear where the 51b n-grams track semantics and context, like 3.8 with its loss of real world knowledge bolted back on. Otherwise? Sparse full-attention with de
TL;DR - A ~4M example, ~3B token CoT dataset designed around helping small models think more concisely, accurately and reliably. Hi all! For the past few months I have been working on a dataset design
v4.5.0: Added native desktop control, agent execution budgets, concurrency limits, loop protection, and safer local embedding fallback. v4.6.0: Rebuilt agents around durable parent-led orchestration.
I’ve been experimenting with a custom inference/runtime research project called CRANE V2, mostly because I wanted to answer a stupid question: How far can you push absurdly large MoE models on an ordi
I am not a huge fan of AI, but my friends like to joke and say I am torturing computers because I use all sorts of virtual machines, commands, and once split in half a 1000 defective laptop (accident)
WeMM-Embedding-9B is a universal multimodal embedding model built on Qwen3.5. It accepts text, images, videos, visual documents, and interleaved multimodal inputs, and returns a 4,096-dimensional L2-n
Upgraded ollama from .32.6 to .32.15 ran a test on my qwen3:30b and it went from 90+ t/s down to 45+t/s System has 4 and v620 pro and one rtx3090 Before it was only using the amd, now it's using all 5
Source of Claims: https://x.com/Andy_ShuoYang/status/2090856976880472439 Your gaming PC can now serve frontier models at interactive speed using official checkpoints without extreme quantization! Qwen
Now, some of you might not believe me from the get go, and it's completely fine, But for those who are willing to give Tap a chance, a whole new world of voice dictation is going to unlock for you. Ta
https://preview.redd.it/usib0obv1blh1.png?width=1171&format=png&auto=webp&s=fa7b6e9155fcc461a2d397f9cf75aece4c72235c https://preview.redd.it/w453mok02blh1.png?width=2613&format=png&auto=webp&s=693182d
after 3 months and 800 burned... Unbounded Labs is proud to introduce Bart, our vintage LLM: 2.82B parameters trained from scratch on 20.1B tokens of English written before 1931. You can talk to it ri
after 3 months and 800 burned... Unbounded Labs is proud to introduce Bart, our vintage LLM: 2.82B parameters trained from scratch on 20.1B tokens of English written before 1931. You can talk to it ri
My test is running DeepSeek v4 Flash 0731 at native size on DGX Spark machine (GB10, 128 GB unified memory). The model size is bigger than RAM, so weights will be loaded many times when running. To im
Graph: A cumulative metric of Speed x Context Size - the higher the dot - the better. Helpful for looking for the most balanced solution. The research relies on DFlash2 Q2 work by AnalogAlok: https://