Search: “r-localllama”

How far are we from a local model being able to optimize it's own runtime stack?

A lot of projects have come up recently consisting of specific hardware config specialized forks of existing conferencing engines, but they all seem to be made using Claude or OpenAI's biggest frontie

13 Oct 2026

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Weird blog post from OrcaRouter

DGX agent

I came across this blog post from OrcaRouter: https://www.orcarouter.ai/blog/qwen-3-8-next-vs-qwen-3-8 Not only is the blog post nonsensical in some parts, it's also completely without any value. It b

model-releasesr-localllama

13 Oct 2026

A 'free' multi-modal 'embedding' model (already) included with your LLM: vLLM plugin + upstream PR

Disclaimer: I work for NuMind, the company behind this plugin. TLDR: Text/image (simili-)embeddings from the LLM you already have loaded :-) I've been working on making it easier to obtain embeddings

A Record of the Process of Increasing MFU from 21% to 50% While Pre-training 21B MoE Using a Single GH200 Chip (Includes Code and Logs)

I am an independent researcher. Due to budget constraints, I shifted my 21B MoE pre-training—which I had originally planned to run on 64 H100s—to a single NVIDIA GH200 ($2.29 per hour), and I have com

Boole-1.7B: A 1-bit Qwen3-1.7B model that outperforms Bonsai-1.7B

We're launching Precision Numerics by releasing Boole-1.7B, a 1-bit version of Qwen3-1.7B. 247 MB packed weights: 13.9× smaller than BF16. 85.9% of the BF16 model’s average score across our 10-task ev

BrokenML: can agents fix broken ML jobs?

Write-up: https://zhebrak.io/posts/brokenml/ Repo: https://github.com/zhebrak/brokenml I built a benchmark to determine the access level agents need to successfully fix broken ML workloads (PyTorch ov

Decision 3.0: open, multimodal decision models from 0.6B to 27B. Ranks #1 on both Jev Decision Index boards

With the vLLM Semantic Router team we released Decision 3.0 (d3), our new SOTA decision model family ranging from 0.6B to 27B, Apache-2.0, all reading text, images and video. How it works - Built on Q

Explore DNA with DeepMind’s AlphaGenome and in-browser MedGemma 1.5 4B via WebGPU

Hey everyone, Sharing an educational tool called AskCyph Health Explorer to explore how single-letter DNA mutations lead to disease, how they alter 3D anatomy, inheritance probabilities, and available

graphsignal-top: a `top` for inference GPUs in the open-source Graphsignal profiler, kernels ranked by time, live, with readable names

A few weeks ago we posted the open-source Graphsignal profiler here: wrap vllm serve / sglang serve / whatever in graphsignal-run and it serves everything it measures as JSON, mainly so an agent can r

I made an open-source AI leaderboard that shows the best models your machine can run

Hey everyone, I made whatmodel.sh To use it, Enter your RAM and GPU See what models you can run locally with its rank and the command to run it Compare them with API models on the same score Building

Improve my vibe-slopped pipeline

tl;dr: Are there other models I should try? Is there something better than Kimi K3 for a code review? Is GLM-5.3 better at finding/patching security vulnerabilities? Are skills like https://github.com

Lecture: Building a GPU Compiler From Scratch. Lowering PyTorch Graphs to CUDA via Multi-Level IRs (GPU_MODE video + article)

https://preview.redd.it/3ekngxk224vh1.png?width=1080&format=png&auto=webp&s=9bfe5c92033e17d5b2fb4529f1320c99455a138b I built a GPU compiler and wrote an article about it, focusing on concepts shared b

llama.cpp optimizations for specific hardware and models

I've made a few posts recently, just talking about the hardware I've got and the model I use. Pair of P100s, Unsloth Qwen3.6 35b A3B UD Q4 K XL. There's been a ton of talk of model and hardware specif

LLC: lightweight Open WebUI alt now with Agents+MCP+RAG+Image Editing

Quick context: LocalLightChat (LLC) is a chat frontend for local LLMs. Download it, run it, that's it. No installation (unless you want to), no dependencies, and it runs on pretty much anything, inclu

LSearch - Spotlight for windows with embedding

I built a spotlight search for windows. It has an embedding model and a MoE models to go with it. The MoE It's 2 Qwen 3.5 4B models, I fine tuned one for coding work and another for agentic work, and

openbmb/MiniCPM-V-4.7-1B

Gguf's: https://huggingface.co/openbmb/MiniCPM-V-4.7-1B-gguf 'MiniCPM-V 4.7 is the latest model in the MiniCPM-V series, available in a 1B variant. It builds on the family's single-image, multi-image,

openbmb/MiniCPM-V-4.7-35B-A3B

gguf's: https://huggingface.co/openbmb/MiniCPM-V-4.7-35B-A3B-gguf 'MiniCPM-V 4.7 is the latest model in the MiniCPM-V series, available in a 35B-A3B variant, with 35B-A3B being the first MoE model in

QLoRA on ExLlamaV3 Quants - Train @ 2-8bpw

EXL3-QLoRA I've built this over the last few months and I think it's stable enough to share now. A fork of ExLlamaV3, this is a native ExLlamaV3 training implementation with plain PyTorch, without tra

Qwen 3.8 Flash Next Q3 S - Benchmark creation

After running a benchmark test on Exo in Opencode and being amazed by the result, I thought I'd try other models on Opencode. Many were surprisingly terrible. Especially some mid level models. Earlier

Qwen released QwenGyre - RL framework for training AI agents on extremely long complex tasks

Introducing QwenGyre, bringing elastic RL to extreme-long-horizon (XLong) agent training. It launches with two core designs: - Elastic Scheduler: reallocates GPUs between rollout & training as demand

Qwen3.8-27B vs Frontier vs Others

Here're results from my final test of coding, coding (panda/numpy), data analysis, data-driven decision-making, and local RAG domains I interact with most often. This test, which is refined from a 500

Qwen3.8-Flash-Next on six 3090s with vLLM: 7 users at 262k, 6.5k t/s prefill, 57 t/s decode solo / 206 t/s aggregate with all 7 generating. Fixes for the upstream included.

Write-up, serve command and raw data: https://github.com/lukolszewski/vllm This shows vLLM fixes and setup that let you put PLE on SSD and save 108GB of system RAM, use int8 kv and squeeze more users

Receipt scanner that runs a fine-tuned 0.8B vision model on the phone (free) - detailed cost tracking

I wanted to track grocery spending per item without sending my receipts to anyone, so I built an Android app around a small vision model that reads a photo of a receipt into JSON: store, date, total,

Running unsloth/Qwen3.8-27B-UD-IQ4_XS 262k q4_0 KV + vision on 16GB VRAM @ 80t/s and 1500pp

I get a lot of hate whenever I tell people I run a Q4 quant of Qwen3.8 27B with 262k KV 80t/s with vision, all while running 2 screens on windows on the same card. People seem to think I am lying or m

strix-llama new branch: prompt processing +21–54% over Vulkan and up to 2.7× over stock ROCm kernel at 128k, same outputs quality

I've been tuning llama.cpp on my Strix Halo box (Radeon 8060S, 128 GB) and the numbers are good enough to share. It's a branch called rdna35-perf on top of halo-box/strix-llama.cpp, a llama.cpp fork a

VectorDash: A full Windows search replacement incorporating Embeddinggemma-2

Inspired by DigUp for macOS, VectorDash aims to be a full Windows 11 search replacement, powered by EmbeddingGemma 2. Full indexing: search by meaning or exact words across photos, screenshots, audio,

What are you doing for a reliable, well designed local model based AI search?

Ive been using OpenWebUI's inbuilt web search and I can't put my finger on it, but it's not quite as polished, reliable, seamless as using Claude or Gemini. And its often slow beyond a limit that make

2x 3090, 3x 3060 potential (smart and informed) future upgrade paths (send help)

I'm trying to figure out the best long-term upgrade path for my setup. Right now my GPUs are spread across several consumer PCs: Main: 5900X, 32GB DDR4, 2× RTX 3090 24GB. Secondary: i3-10320, 16GB DDR

Building a 4x R9700 setup for a 10 person startup

Just wanted to share a build I am doing for a client. $18k. Specs: Threadripper 9970x 128gb DDR5 ECC 5600 4x AMD Radeon R9700 AI Pro 32gb(128gb total VRAM) 1600W PSU(GPUs are undervolted and under 210

Converting dense models into Mixture-of-Experts

For the past few weeks I've been trying out converting existing dense models to sparse Mixture-of-Experts models, with no pretraining from scratch. The Idea If you can turn a dense model into an MoE t

Every Model That Can Be Run On 10-16GB VRAM Ranked

It's been 4 years since c.ai first hallucination model, and yet, we're nowhere good enough at LLMs in terms of spontaneity/interesting hallucination features. Fortunately, last year brought new archit

Final revision of zzpanic vllm-radiance for 1 x R9700 GPU, qwen3.8-27b

Summary of project status metrics Since StillDeadcode has released his new radiance inference engine, the writing is on the wall for the old vllm-radiance builds, people have moved on to newer and bet

Gemma4 E4B EXL3 - Alternative to Llama.cpp on Jetson Orin

My Jetson Orin–optimized engine, little-gemma V1.0, substantially outperforms llama.cpp. Even after exhausting every practical GGUF option, however, it still falls short of the ideal performance level

I built a lightweight tool that automatically frees up VRAM for gaming and restores your local LLMs afterward

I run local LLMs pretty much around the clock, but I also enjoy gaming on the same PC. Having models sitting in VRAM while gaming isn't exactly ideal, and constantly unloading and reloading them manua

I ran Qwen 3.8 Flash-Next on my AMD 7900 XTX at 500k context. All local and what a shift it has been.

I was running Qwen3.8-Flash-Next (base and later Swift-1.5 NVFP4) on a paid pod seat for about a month, and it started answering with short strings that echoed its own earlier reasoning. I assumed the

I tested different Qwen 3.8 27B quants

There is a lot of discussion which quant to use. Especially a lot of post how much speed you gain with FP4, but not much about quality. So i tested some of the models that I use on my R9700. Models: Q

I trained a 102M recursive BitNet-v2 model from scratch: 64K context, trained on less than 5B tokens

DISCLAIMER: the post and the model card was made with the assist of AI. Hiya, I’m releasing Recursive BitNet N-Gram 102M, a small experiment combining ternary weights, shared transformer layers, and h

local semantic file search for Linux (Rust, llama.cpp, EmbeddingGemma 2)

indexes PDFs, Office files, scans, screenshots, audio and video, and finds them by content. A query like 'when did they decide to freeze hiring' opens the recording at that second. * EmbeddingGemma 2

M5 Ultra 256GB or 2x DGX Spark? Which one?

Currently contemplating adding either M5 Ultra 256GB or dual DGX Spark in addition to existing 5090. I have been researching for past few days and have zeroed in on dual sparks because higher throughp

microsoft/AesCode 8B and 32B

https://preview.redd.it/r2ledkyuntuh1.png?width=1684&format=png&auto=webp&s=9ed4518b2d0f20f44835ebbfac605581da186a57 AesCode generates information-rich visual artifacts such as slides, posters, and da

Now you can grow Bonsai on your potato

No more excuses for GPU-poor folks not to start LLMing! Full 27B-class reasoning in ternary transformer weights, for llama.cpp (CUDA, Metal, CPU) ~9.3x smaller than FP16 (ideal) | 98.2% of FP16 intell

OMG! If you have a Mac with 64GB, try Qwen3.8-Flash-Next-oQ4e-mtp with oMLX!

I'm genuinely shocked! I was able to run Qwen3.8-Flash-Next-oQ4e-mtp on M3Max 64GB with oMLX! Even a couple of weeks ago, I wasn't able to get it to run. I just tried the latest commit for fun, and it

OpenMed 3.0 is out: Apache-2.0 clinical AI that runs fully local and never falls back to the cloud. 422 open issues if you want in on 3.1

Hey r/LocalLLaMA! Maintainer here. Some of you might remember OpenMed from our 1.8 post a few months back. Quick recap: it's an open-source (Apache-2.0) medical AI toolkit with one rule we never break

Qwen 3.8 27B Q5 vs Qwen 3.8 Next Q3_S for document analysis

I have an app that processes ~500 different documents every day. An agent analyzes them, sorts them, tags them, extracts various legal nuances, creates descriptions, etc. So far I've been using Qwen 3

Qwen3.8 Flash Next fixed my GNOME extension

I love Dash2Dock Lite, but Icedman is always a week or two before updates. I randomly ran pacman for the first time in weeks, not realizing GNOME had a major update. I told it to fix it in DeepSeek Ha

Running Next Flash IQ3_XXS at ~70 tok/s with 100k context or 2 instances of Qwen 3.6 35B A3B IQ4 at ~145 tok/s with 256k all on $500 of ex mining BC-250 boards

This will be my third update on the bc-250 cluster and for my first forray into local ai I have been having a blast. This has turned from a weekend experiment into something I think I will use in the

'Strata' for GLM5.3 Flash is here for some! Project Maya

I stumbled across this as I was currently having glm5.3 flash run only around 10tok/s basically unusable. With Maya i am running 30 tok/s now. GLM5.3 feels even with this low quant like a much more en

UPDATE: Qwen 3.8 27B 140 tok/s on single RTX 3090 Megakernel: KL divergence 0.0009 vs llama.cpp

This is a follow-up to my post from yesterday (https://www.reddit.com/r/LocalLLaMA/comments/1x2erdj/qwen3827b_on_a_single_3090_140_toks_on_code_with/) detailing the CUDA megakernel. Thank you all for