Knowledge catalogue
Theoretical use case: I have a compressed version of wikipedia, and some function or tool fooRAG. Then I set my system prompt (or make a skill, or hook, etc.) to include 'You can search wikipedia with
Qwen 27b Q3_K_M 2x rx 580 8gb (~50 each in my country, edge cases 60 per gpu) gives us 16gb vram We used it on an old already existing ddr3 motherboard with 2 gpu slots(you can buy it ror around 200$
Following up on my previous post about my budget server setup (Intel N100 + RTX 5060 Ti 16GB), a few of you asked for a deeper dive into my actual inference config and real-world agentic performance.
Hello all, I'm fairly new to the whole Llm space and could really use some guidance from people who've done similar work. Im building a pipeline to extract structured data from large-batch of invoices
If you're running Ollama, llama.cpp, or vLLM behind nginx or HAProxy for more than a single user, you've probably hit this: nginx treats a 10-token prompt and a 10k-token prompt as identical 'one requ
Guys, could someone smarter than me explain what makes Deepseek Harness so efficient? I run it with local Qwen 3.8 (Q6). I tried Opencode/Openchamber (my favourite so far), Pi agent and Hermes. New Qw
Note: I use a Mac Studio M3U with 512 GB RAM, so this is not for everyone. I have been using antirez/ds4 with DS4 Flash for a few weeks now. Top quality, I am really impressed. This combination just d
EXL3 is an alternative to llama.cpp. And while there is extensive tooling for llama.cpp, EXL3's primary deployment (TabbyAPI), has a OpenAI compatible API so it shouldn't matter. Why won't this tool m
Do you think today’s GPU-poor systems running Qwen 3.5 9B can outperform the ones we had with ChatGPT-4o? I mean, when 4o disappeared, people felt like they’d lost a great model, and now Qwen 3.5 9B Q
Trying to get a feel for where I stand. If you can list your relevant hardware and model used, that would be awesome. Here's mine: Model: Qwen3.8-27B-heretic-ara, Q5_K_M GGUF T/s: ~30-32 t/sec (I thin
Original Article - https://x.com/p_nawrot/status/2089315591010079034 I've spent the last few years working on efficient attention and KV Cache Compression. I've read many papers, dug deep into referen
I created 3 SVG prompts, each one rather hard. Perspective: A animated drone view perspective on a park. Beauty: A beach scene with an evil cat Composition: An AGI breaking out of a virtual sandbox pr
I'm back. Yesterday I released the first version of hyper-optimized Qwen3.8-27B inference engine for a RTX 3090, reaching 82 tps on single request and 672 peak. Over the last 24 hours I've been explor
Support for the new ling 3.0 models has been merged into llama.cpp: https://github.com/ggml-org/llama.cpp/pull/26608#event-29549472828 Ling tiny 8b1b - https://huggingface.co/inclusionAI/Ling-3.0-tiny
In medium reasoning mode, it both scores higher than the 3.6 version, AND is very much more efficient (almost half requests needed, and a third less tokens generated) - at DeepSeek v4 Flash 3107 MXFP4
Hi all! I finally just got around to testing out Qwen 3.8 27b. I'm using Unsloth's UD-Q8_K_XL quant as a sit-in replacement to Qwen 3.6 27b, same quant size. Wow -- this thing isn't messing around. I
for those looking for something small AND powerful, there is a new 1B (they claim, it looks more like 1.7B ...) model that claims to beat qwen 3.5 0.8B & 2B and gemma 4 E2B on a range of benchmarks. t
Now that OpenCode plan isn't really worth it anymore, I'm looking for other options. I'm trying to figure out the Ollama Cloud subscription, specifically the $20 tier, because that's my budget for mon
I’m trying to compare Ollama Pro with OpenCode Go, specifically in terms of how much real-world usage each plan provides. Ollama Pro costs 20/month, while OpenCode Go costs approximately 10/month. Bas
I run locally Ollama + Qwen (tried multiples versions) and I connected it with MCP into my local SQLite database. When I ask a question, 90% of times I see it thinking and just stop and never answer t
pagedMark removes AI provenance from content you generated yourself. Two different things, and it is worth separating them. The first is metadata: C2PA Content Credentials, EXIF, XMP, IPTC, the genera
We introduce Mobius-v0, an architecture that comprises a globally shared Memory (FFN) that stores knowledge vectors and multiple Reasoners (Self-Attn) that iteratively achieve compositional reasoning.
Thought i would just add my own flags here for llama.cpp (literally pulled and rebuilt latest today). Running on a 4090 FE and 48gb of DDR4 ram on WSL2. Basically you have 3 knobs you can tune for max
Yes, it sucks to waste time waiting on 16K+ reasoning tokens alone. But here's the thing, this is only a 27B model trying to perform on par with 1T+ parameter models. Something has to be sacrificed, a
So many comments begging for these models, and I get it. People from the labs frequent r/LocalLLaMA, so maybe the begging comments aren't useless. They make demand known at least. Same with polls, etc
Artificial analysis index scores Qwen 3.5 27b: 35 Qwen 3.6 27b: 38 Qwen 3.8 27b: 52 What the hell kind of a jump was that? Even if it is benchmaxxed, the jump is insane. Qwen3.6 35b A3b: 32 That's ~6
Sorry for the slop, but I was impressed by this model as I have been testing Qwen3.8-27B Q8_0 locally on my ROG Flow Z13 (Ryzen AI Max+ 395, 128 GB unified memory) and this model was the only one who
Overview EVIE-Preview-4.5B is a state-of-the-art multilingual Visual Document Retrieval (VDR) model built upon Qwen3.5-4B. It employs ColBERT-style late interaction with native 128-dimensional multi-v
Yes it uses a ton more reasoning tokens than 3.6 did But test in on the same tasks with the other chinese models, glm 5.3, deepseek v4 flash and pro, etc it's really similar, and they are needed The r
qwen3.8 27B has seriously impressive benchmarks on its model card, but that's for the unquantised version. Almost everyone here will run one quant or another. Are there good benchmarks for how those q
There’s a hands-on workshop on August 29 that builds and benchmarks this properly, end to end, using entirely open models, no API calls involved. Led by Ben Auffarth, AI Consultant and Founder of Chel
I don't understand this. Deepseek used half as many requests as GLM 5.2 for 3 times the usage, and I'm getting 10 times the requests from flash for like 1/100th the usage. I know Pro is supposed to be
Just curious - I like to be able to see and manage the scripts my agent is working on. I find stuff like Claude Code and Open Code useful for doing stuff on my linux box but I don't understand why peo
I've got a dedicated AI rig sitting here with a RTX 5090 and 96GB RAM and for the past few years have been using Windows 11 and primarily LM Studio, but have also used vLLM, llama.cpp and Ollama. With
I've pulled two models, `gemma4:latest` and `qwen3.5:latest`, and tried to use them in Android Studio and Itellij. I've followed the docs for both IDEs and they show in their chat features. My problem
Title pretty much says it all. I’ve deployed Qwen 3.8 27B using vLLM on an RTX 6000 Pro (tried multiple vLLM releases and launch recipes), but I can't get it into a usable state because of crazy long
Including the rationalisation for the data below - this is a more robust version of an earlier post I did similar to this - explaining below: How I chose the comparisons The basic question I’m trying
Ollama sucks in terms of modaltiy options , model variety and is better in terms of usage limits . Is there any other alternative that could provide the same usage limits with better modality options
Running a 12700k + 5060 Ti 16 gb with 128 gb DDR4 ram and I'm wondering what's the ideal setup to maximize performance. I did some preliminary stuff but I have to admit, I'm still learning and kinda j
Hey Ollama community, If you are planning to fine-tune and build your own local Turkish coding models (using Unsloth / LoRA to export to GGUF and run in Ollama via custom Modelfiles), I've open-source
I was surprised by the fact that the qwen 3.8 27b download count is about 1 million (globally). This means that even on this subreddit, very few people have used 27b. At most 50k–100k active users, an
At hugging face, this is the model I use for most of my analysis that most of my services have to not be refused. previous versions are quite good. It seems this might have dropped today and pulling r
I don't know what's happening. So, when i use models in openwebui everything works, ollama uses the GPU fully, everything's perfect. Then, I use Hermes, and loads the model in VRAM (fully, no offload
I'm just thinking, ever since microsoft announced bitnet, this sub (and myself) has been hoping for massive ternary models. In the last month alone, prismML dropped 27B ternary (though I've read commu
Single prompt 'make me a 3d scene of an anime girl' (stolen from someone on discord, I like it a lot since it seems like every model struggles a lot with it) Everything is one shot. No vision, no iter
With all the new models released recently one important upgrade went unnoticed: Llama.cpp bumped ROCm from 7.2 to 7.14. I was waiting for that because in 7.14 support for gfx1103 (Radeon 780m) was int
Hey! Like a lot of people here with consumer GPUs (RTX 4060 8GB in my case), I wanted to see if I could use local models for daily coding tasks instead of burning cloud credits on simple boilerplate.
Honestly I am surprised at the speed and performance I am getting from the model, it does not allow for complete hands off like Claude 5.0 but that is not what I want, i want to be able to iterate and
We currently have an ancient qwen 3.5 27b on the cloud probably and that sucks compared to later versions , why doesnt ollama update automatically . We have practically ancient models right now on the
In this commits, the 35B model was removed. Looks like it's confirming the 35B model won't get released. I think they need to be made aware how big the 35 moe is widely used. Think need to make noise
I’m running Ollama on an M4 Pro Mac with 48 GB unified memory and testing local coding models for repository work. Direct Ollama inference is fine. Qwen3.8 27B-MLX runs well enough for me, and the loc
If you are using llama-server with their web-ui for testing, keep in mind, that the reasoning selector is just a reasoning budget aka a hard cap and has, at least to my knowledge, nothing at all to do
https://developer.nvidia.com/blog/serve-qwen3-8-2-4t-a95b-a-2-4t-parameter-model-with-configurable-reasoning-on-nvidia-gb300-nvl72/ 4k tokens per second per GPU of which there are 72. 350 tokens per s
Hi All, I have a MacBook Pro M5 Max with 48 GB of unified memory. Also have a M4 Mac Mini 16GB unified memory. I want to setup local AI and I heard about QWEN 3.8 model. As I want to work on a repo th
https://preview.redd.it/wkg27e152qjh1.png?width=853&format=png&auto=webp&s=2e3f8b11ea6393041f501e95c5835f9bea0245dd So ive been trying different harnesses and coding agents with the new qwen 3.8 , and
Hi, After a long night of optimizations, I believe I have made the fastest inference engine for Qwen3.6-28B on a 3090. Quick metrics: - 250w power capped - Up to 195k context (ships with 150k for safe
Did a quick local test because I wanted to see what is actually usable on my 12GB laptop GPU. I tested the newer Qwen3.8 27B dense files at Q2 and Q3, then compared them against Qwen3.6 35B-A3B MoE. H
I did a short test of the different reasoning efforts, since on default xhigh the model thinks a lot. Not very scientific, just a quick 'generate an SVG of a pelican on a bicycle' prompt with 3 differ
one of my llm hobbies is re-creating graphics demos i used to write in BASIC in the late 1980s. i slopped together an agentic harness and a basic-to-js transpiler in a web page i've been playing with
My local inference build, with: ASUS WS C422 PRO/SE 10-core Xeon W-2255 64GB ECC RAM 64GB VRAM Pimped case with TurboLEDz indicating the frequencies of the 10 xeon cores. Running llama.cpp with SYCL b