Search: “r-localllama”

Qwen 3.8 27B Vs. Qwen 3.6 27B on oMLX

Quality: 81.1 → 87.7 (+8%) Speed: 35 → 29 tok/s (−16%) Runtime: 8m51s → 44m39s (5x longer) Output tokens: 18K → 78K (🤯) Noticeably better quality, but you're paying for it with tokens and time. Full b

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Qwen-3.8-Next-Flash Ngram Hot-Swappable Knowledge Injector for llama.cpp

DGX agent

Looking into the new Qwen architecture, I was curious if you could modify the Ngram PLE Table to make it work like a long-term knowledge database. It turns out that, with some limitations, you can. I

model-releasesr-localllama

3 Sept 2026

Qwen3.6 35b Q2_XXS: Being GPU poor in 2026 is not so bad

A potato can create a very cool RPG in 24 minutes Laptop I3 8gb RAM 0gb VRAM, Windows 11 llama-server.exe --host 0.0.0.0 --port 8080 -m modelsqwen3.6-35b-InstructQwen_Qwen3.6-35B-A3B-IQ2_XXS.gguf -c 8

Qwen3.8-Flash-Next MTP merged in ik_llama.cpp (integrated head or separate -md file)... 45 → 90 tok/s on a 5090 + 128GB, works down to a 12GB 4070

ik_llama.cpp merged qwen4exp MTP support yesterday (PR #2369, mine, reviewed and tested by four other people on their own hardware). It's on main now, no fork or patch needed. Posting since the last c

Qwen3.8-Flash-Next on 2x3090 + DDR4: 17 → 25-29 t/s decode with the expert cache PR

Sharing some numbers because most posts on this model are either using a single 3090 or unified systems from what I've seen. My current setup: 2x RTX 3090 (PCIe 3.0), dual Xeon E5-2696 v4, 188 GB usab

[RELEASE] - SupraGDN-5M - a tiny GatedDeltaNet model competing with other community while models trained on MUCH less data!

https://preview.redd.it/5rs3d7vtgbnh1.png?width=950&format=png&auto=webp&s=df828c1146bc6167bd09d3300018aba426fc1ab2 Hey, r/LocalLLaMA ! We are releasing SupraGDN-5M. It's a tiny GatedDeltaNet (GDN1) b

Repodify, a fully local & opensource podcast summarizer, or BYOK if don't have GPU.

Disclaimer: I'm the builder. --- Over the past 2 yrs, I was working on a SaaS ML project & got very interested in ML/DL/AI. As everybody else, there were some normal paths I took to build a solid unde

Testing Deepseek-V4-Flash-Vision-EXP on Dual RTX6000 build

Upgraded my work LLM server from DSV4-Flash-0731 to the new Vision-Exp model and did this little test for fun to see quality comparison. Test was performed in Openweb UI connected to my vLLM container

Training-free hack to reduce MoE reasoning tokens by 8.5% without losing accuracy but increasing Active parameters per token in MOE (Qwen 35B A4B+)

Hey everyone, Wanted to share a short paper we just published exploring a simple but surprisingly effective optimization for sparse MoE reasoning models. The problem: MoE reasoning models (especially

We built an open-source, model-neutral agent harness and compared it with claude managed agents - for the same model, got same accuracy, upto 75% lower cost

We have been working on an open-source, model-neutral agent harness for general purpose agents called TrueForge, and wanted to understand how much the harness itself actually matters. So we ran 14 tas

We need a better taxonomy for what people are calling 'continual learning'

Continual learning isn't some fake term but a real goal and arguably one of the most important open ones in 2026. It roughly means: ‘building systems that keep acquiring useful knowledge or skills aft

What are your best sources to learn about LLM inference?

The job of doing inference is not the same for every model ou there, as it depends on the architecture and configurations and the available machines. Many times we would not have the right machines to

WVY is a handwritten language model. Every response was written by one person to demonstrate that the illusion of intelligence is not exclusive to parameter count.

***This screenshot is an app i made for creating a dataset from scratch, this is not a real chat with the model*** First of all i want to shout out everyone that actually tested our work .. we got 500

Android Studios native Gemma 4 runs on llama.cpp

https://preview.redd.it/6e9xb57a42nh1.png?width=787&format=png&auto=webp&s=ffae7996bbf8ab00498cc62c733e7597dc550f24 I'm not sure how many people care about Android Studio, but I think it's cool that G

Confirmed bolting Q8 NGram into IQ4 Qwen no speed degradation

This came from another thread or comment. I forgot exactly where, but the basic idea was to replace the 51B N-gram layer in Qwen 3.8 Next with a much higher precision version. Someone running a 5090 r

First local-LLM tuning attempt: Qwen3.8-27B true Q4_K_M at 13.2 tok/s near 50-61K context on RTX 5080 16GB

This was my first serious attempt at tuning a local LLM. I started because Qwen3.8-27B IQ3 was fast on my RTX 5080 but the coding quality disappointed me, and the Q4 profiles I tried in LM Studio were

LocalLLaMA is unironically one of the best places to go to get up to date AI news.

One of the other posts today by user u/Howard_banister confirmed what I've been seeing from the other AI subreddits as well. Most of these other subs are 90% trend hopping crypto-bros equivalent peopl

Looking for a small LLM for Linux command generation

I'm looking for a small LLM to act as a Linux command assistant. I will use llama.cpp. use case: - User asks in natural language. Model outputs only the shell command - Should work well with no reason

Mac Heads: Is there any point to MLX in September 2026?

This may be somewhat specific to Qwen3.8 27b and the Apple M5 series, perhaps, but enough of us are running this combo that it's worth tossing out there. GGUF models with MTP have been the fastest way

Opencode vs Deepseek harness: my experience with Qwen 3.8 27b

Anybody else struggling with deepseek after the initial prompt? Somehow it is getting mixed up very easily, even button functionality has been PITA when doing stuff. Never had these problems with Open

Perplexity open-sourced their Mac inference server for Qwen 3.6

https://preview.redd.it/6h4xc5o8l6nh1.png?width=1158&format=png&auto=webp&s=b65074b6baaa1faa2347e5259229c8ba803bcd4b Here is link to repo: https://github.com/perplexityai/pplx-garden/tree/main/lily It

Quad R9700 AI Pro with vLLM-Radiance easily reaching 17,6k PP

https://preview.redd.it/74bmvel9b5nh1.png?width=1602&format=png&auto=webp&s=0d0c1adaa016a486ffd97c4c466e980dc611b139 I've only recently started looking deeper into vLLM after running llama.cpp for a g

Qwen will be the king?

Extended reasoning and post-training appear to be the keys used by DeepSeek, Qwen, and GLM to boost performance (leveraging higher token counts). And Qwen 4 hasn't even been released yet. Of course, w

Qwen3.8 Flash AP Quants

Quite surprised to be beating other high quality quants. It took a lot of benchmarking to get here and we are quite pleased with these, hope they are useful to the community. It required a modified wa

Qwen3.8-flash-next sees corruption everywhere

Hi, I've noticed that the model often sees 'garbled text' in its context. Sometimes it declare that the tools instructions are corrupted, sometimes it is the content of some .md files, ora other files

VoxGen, an AMD-optimized TTS inference engine for VoxCPM 2 models

Hi, everyone, I’ve just released VoxGen, a lightweight native inference engine for VoxCPM2, written in Rust and using Vulkan compute instead of Python/PyTorch/CUDA. Why VoxGen? The main reason I start

Which one will you choose and why, between R9700 32GB vs W7800 48GB?

I'm planning to upgrade my workstation (linux with 5700X/64GB DDR4) for local inference and pytorch training. I'm trying to decide between: 2× AMD Radeon AI PRO R9700 32GB 2× AMD Radeon PRO W7800 48GB

Your favorite fastest abliterated/safety removed 3.6 and 3.8 27b?

Not written by AI all mistakes mine. I saw people on the subreddit saying that 3.6 works better without thinking. It made me want to know for certain about which is better, 3.6 or 3.8 for low thinking

A very confusing report from Puget Systems

Just to name a few: running Qwen3 8B on a 32GB GPU running Qwen3.6-27B Q4_K_M on 2 x R9700 quote: 'each prompt was sized at 500 input and 500 output tokens' for a full system that costs $18,775?? I do

All currently popular local models in one table + Opus 4.8 results

If you are thinking what model will fit best your HW specs and tasks you are doing here is one table with all currently popular models that still can be considered as local. LLM Test Scores Feature De

CMP170Hx “Spark” Machine

I got the CMP170 cards and unlocked them. I wanted to share my set up for CUDA since maybe it would be useful to others. First off, I hate e-waste and we are in a special time for RAM. I wanted to hav

ExLlamav3 Recent Updates : CPU offload, GLM-5.3-FLASH, Qwen3.8-Flash, SC Quants ++

More new massive updates from turboderp: - CPU offload of MoE experts - Qwen-3.8-Flash-Next ngram disk offload - GLM-5.3-Flash - New self-calibrated optimization technique - Countless other optimizati

Hardware

Given how common RTX 3090 use is for LLMs, why don't we see more INT8 W8A8 models ?

Based on https://huggingface.co/hardware, the RTX 3090 is the second most used GPU by LLM enthusiasts. Because RTX 3090 has native INT8 tensors cores, it can provide better performance with INT8 W8A8.

hardwarer-localllama

I pushed Qwen3.8-27B to 2.000 prefill per second and 132 decode per second on A RTX 3090.

Yoyo I'm back with updates to the fastest inference engine with minimal quality loss for Qwen3.8-27B. The last few weeks I've been optimizing decode speed and I don't think it can be pushed further, u

Kaitchup posted Qwen3.8 27B Benchmarks for quants from Q4 to Q1

Kaitchup just posted results of his benchmarks for Qwen3.8 27B for quants from different labs, Q4 to Q1, . All the details are hidden behind the paywall, but high level result is visible and looks lik

Hardware

Multilingual Tiny (3.7B) Reasoning MoE pretrained from scratch on a consumer-grade GPU

Hello! I've just uploaded a recent checkpoint of my model trained from scratch: https://huggingface.co/piotr-ai/polanka_3.7b_exp_wip_260901 It was pre-trained, mid-trained, and fine-tuned on a single

hardwarer-localllama

New Model: Spark-X2.5-4B, Spark-X2.5-1.7B

I was browsing HF for small LLMs and run into this model. It does not seem to be a fine tune - the model has its own architecture. https://huggingface.co/XHToken/Spark-X2.5-1.7B https://huggingface.co

Question: Why is prefill unbelievably faster in vLLM than other inference engines?

I only started using some vLLM forks recently in a 4 x 48GB 4090 system. DS4F - ~5000pp/180tg (DSpark) Qwen3.8 Flash next - ~7500pp/135tg (MTP) This is amazing, like having the API in my house. But it

Qwen 3.8 27b (Q4KM) oneshot a Super Mario clone

I am absolutely blown away. Yes my setup is crap but the fact that it managed to do this in a single take is unbelievable (and I'm a developer). Hardware used: - Windows PC with 4070ti (12GB VRAM, 32G

qwen4exp fixes in llama.cpp

if you are on Qwen Flash Next make sure to update your build often https://preview.redd.it/zoq3j9gnjvmh1.jpg?width=900&format=pjpg&auto=webp&s=bb21bc405696df94fd5ce38c22f4ca6dc596135f merged already (

Update: llama.cpp for Radeon VII / MI50 / MI60 — +14% PP, +9% long-context fill vs upstream + adaptive Flash Attention

I posted a new gfx906 based llama.cpp fork a few days ago. One of the main points of critique was that i did not provide sufficient numbers for the gains to be achieved. -- TL;DR: After switching our

Vellium v1.1.0 — Live voice, local STT/TTS and easier llama.cpp setup

Vellium is an open-source, local-first desktop app for AI chat, character roleplay and long-form writing. Recent updates have focused mostly on making local voice and model setups easier to use. Live

Which current local models that can run within 128GB generate the best SVG pelicans?

I used a famous Simon Willison's pelican riding a bicycle prompt on the biggest local LLMs that can run on 128GB Apple Silicon. U used quantizations by Unsloth. Qwen3.8 Flash-Next gives a lot of detai

Compact Rollback MTP: a MTP version for QWEN models for those with little vRAM

I've made a modification of llama.cpp MTP for people that want to run models like QWEN 27B on 16GB and similar setup, the focus is reducing the memory cost of MTP allowing more speed for less ctx cost

CUDA: extend MOE fusion to specdec, earlier MOE glu fusion and topk-router fusion were restricted to 1 token by ynankani · Pull Request #27621 · ggml-org/llama.cpp

I haven't had a chance to test it yet, but it looks very promising. It seems to speed up MTP for MoE models across different draft widths (especially greater than 1). Check the benchmarks. submitted b

Does anyone have real experience with Ornith-1.5-9B for coding

I'm very happy with Qwen-3.8-27B, I run it on my work machine and switched for major part of real coding tasks from cloud subscriptions to it. I use one dedicated headless RTX 3090, and get maybe 1000

Hardware

Doesn't this look like NVIDIA is price fixing?

According to this article Samsung has locked up the 70% of it's future ram production in contracts to companies like Microsoft, Google, and Nvidia. Everyone knows this is driving the ram price increas

hardwarer-localllama

GLM 5.3 and GLM 5.3 Flash ran locally on RTX PRO 6000 WS and built a penthouse using BlenderMCP

I keep seeing demos of AI agents building scenes in Blender through BlenderMCP, so I tried it myself. I ran both models locally for this and picked the GLM 5.3 family(Q4 quant) because videos of it do