Search: “r-localllama”

Zer0Fit - Zero-shot predictions, classifications, and regressions using Google ML research models running locally as a dockerized MCP

AI grad student here. With all the recent interest in Jev, I thought I would share something I built la few months ago that brings ML models and LLMs together in a different way than Jev does for diff

27 Sept 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

2400cc Inference Racer: Dual RTX 3090 motors, NVLink turbo, naked 7840U ThinkPad ECU, VW Golf radiator

Today I present a fine piece of engineering, carefully assembled inside a custom chipboard chassis: the 2400cc Inference Racer, a.k.a. my winter heater. Power comes from two second-hand AORUS RTX 3090

85 GB DeepSeek-V4-Flash at ~3 tok/s on a 12 GB RTX 3060 + 64 GB DDR5 RAM - Overspill for FreeToken, inspired by Colibri

I've been experimenting with ways to run MoE models that don't fit comfortably in RAM, and I ended up making Overspill, a disk tier for FreeToken. The basic idea came from looking at how Colibri handl

Hardware

Are there still any hidden gem gpu's left?

you read about gpu's like the Tesla P40, V100, AMD M150 etc that people pick up for cheap. probably many others as well. Most are either server gpu's being phased out or mining discards, right? The pr

hardwarer-localllama

At what point do LLMs start bootstrapping themselves and generating the next LLM

I suppose technically if that happens it will signal the true start of a singularity because from that point on progress will be exponential and not dependant on humans. right now you still need huge

DistribAI v2

DistribAI is a platform for distributed training! You can train massive or tiny models across tiny or massive amounts of consumer devices with ease! DistribAI is a platform I've been working on for a

For the longest time I’ve felt this sub should have a pinned section where a detailed post about each model should get featured.

For instance whenever a model comes out, what’s the best engine to run it, the best harness and absolute minimum you need to get same or near same re results that the benchmark of that model claims. A

Getting stupidly good results on my 4x3060ti setup.

About a month ago I was having FOMO and was going to spend coin I don't really have on new graphics cards. Instead of doing that though I decided to spend the money on a new motherboard+cpu and try to

How accessible is local AI actually, and what happens if affordable access to frontier models doesn’t last?

Sometimes it’s easy to forget that this sub and others like it are probably the extreme minority when it comes to this hobby. Most people, I would think, don’t use or can’t afford one good GPU, let al

I added Qwen-Image 2.1 + LoRA support to TensorSharp (GGUF, local inference)

I maintain TensorSharp, an open-source inference engine. It can now run Qwen-Image 2.1 locally for text-to-image generation and image editing, with support for its LoRA adapters. I’ve added configs fo

I built a tiny (332MB) CPU-friendly model for document sorting that actually knows when to say 'none fits' (BeeNara)

Hey r/LocalLLaMA! I wanted to share a small project I’ve been working on called BeeNara Why I built this: I was looking for a way to automatically sort my local documents (invoices, letters, contrac

Improved and fixed template for GPT-OSS (again). Includes preserve_thinking and fix for Unsloth-induced bug

I posted an updated GPT-OSS template a couple of months ago, which was based on Unsloth's version. It turns out that both Unsloth's version (and, thus, mine) contain a very serious bug that can degrad

internlm/Intern-Decision 4B and 0.8B

https://huggingface.co/internlm/Intern-Decision-0.8B Intern-Decision-4B Demo | Model Weights | GitHub Intern-Decision-4B is a multimodal structured decision model fine-tuned from Qwen3.5-4B. It accept

Introducing KoboldCpp Agent (and a plea for help)

Hello r/localllama once again, it's me your kobold concedo Been a few months since I last posted here, and today I have something new I'd like to share. Specifically, KoboldCpp now ships with a built-

Just bought a second 3090 but now I don't see the benefits right now.

Hi, My local AI server consists of 96GB Ddr5 and one rtx 3090. I got plenty of stuff running, like krea2, qwen image 2.1, minimax h3, ltx 2.5, qwen 3.6, qwen 3.8 q4...,got even qwen 3.8 Flash next run

Ling Tiny 3.0 is a glimpse of the future

I've been playing around with Ling 3.0 Tiny, which is an 8 billion parameter model (MoE, 1B active). And I've had a lot of poignant thoughts as a result. Just for fun, I got it running with llama.cpp

My Reading Library: Evaluating LLMs on Android Tasks

Can LLM agents actually get through a day in the life of a normal user? That question got me reading papers on Android agents and mobile benchmarks over the past few months. A few patterns kept showin

Qwen 3.8 flash next is based on Qwen 4 architecture, if the announced Qwen 4 27b is also the same architecture with n-grams does it mean I can actually have faster inference on a single 3090 without tweaking much?

I wish Qwen also released dataset and method to fully train a model ourselves but it is what it is. However, I come here with my stupid question because someone can answer it better. And will the mode

Qwen3.8-27B IQ3_XXS vs Qwen3.6-35B-A3B Q4_K_M

Which one is better for difficult tasks like web scrapping, coding, using tools? Looking for any benchmarks because i couldn't actually find one after quite some digging submitted by /u/Loose_Doubt367

Qwen3.8-27B Q4_K_M on 2x3060

For the last few days, I've been using two computers to run multiple agents. My 4x3090 machine is running Qwen 3.8 27B with parallel=2, so I can run two agents at the same time. My pi instances are ru

Qwen3.8 flash next + exllamav3 + hermes is amazing

I know there is nothing new with what I am saying but I recently started with hermes agent (it’s been a while I wanted to but did not have the time). Qwen3.8fn 6bpw exl3 (from turboderp) on a 6x3090 (

Splash 1.1.0 released, GGUF quants support, MLX import and more

On my M5 Pro 64GB I can comfortably work in an agentic setup with the Qwen3.8 27B model in good quality (Unsloth UD-Q4_K_XL) at a decent speed of 50 t/s. Splash combines optimized kernels, excellent s

4-5 days replacing Claude w Qwen 3.8 Next

Hi, human here with rambling thoughts to share. Feel free to skip Overall, I don’t feel like I’m missing much; if anything. On my hardware(M1 ultra w 128Gb) it’s probably not as fast as Claude but I’v

Accidental discovery? or known method i'm missig? - Q's on compiling in sparse engrams and ideas swimming in my head

I set out to test portable Engrams and accidentally ended up testing compiled external memory instead. Am I onto something useful or reinventing a known idea? I've been building a small open research

Did anyone do a full bench of e.g. Qwen Flash Next IQ4 and Qwen 27b FP8? Here are some

edit: had to change formatting as I’m on mobile noe and the table broke.. I let Codex do a quick eval on Qwen 3.8 Flash Next IQ4_XS (served via vLLM + r9v) and Qwen 3.8 27B FP8 (served via vLLM + Radi

Has anyone benchmarked AI agents against the SOLIDWORKS CSWA exam?

Would be interesting right? Models are starting to score higher and higher on benchmarks like Parametric CAD Bench, but can they pass an actual exam? The Certified SOLIDWORKS Associate (CSWA) exam mig

How are you guys thinking about context now, and building around it?

Not asking for anyone’s secrets of the trade, I’m more curious how people are thinking about context now that newer models chew through huge amounts of it for reasoning. the TLDR: I’m starting to thin

How do you use subagents & multiple agent with local models, and how many?

Running qwen3.8 27b nvfp4 on vllm at max context only gives around 8 agents with 32k context each. That doesnt seem like much; what use cases do people use multi-agent frameworks and find it helpful f

How long can I expect to wait until the local ~30B A3B frontier catches up to GLM 5.3 Flash quality?

The jump from Qwen3 Coder 30B A3B to current-day Qwen 3.6 35B A3B is crazy, especially with all the fine-tunes, and that was around 6 months (I didn't care for local AI back then, or AI at all, apart

How would you go about using multiple models together from a singe router (?) or a end point ?

I have 3 machines, My main one can run Qwen3.8 Flash Next at 13-15 tps, i also have an MacMini 16GB whcih can run Orninth 9B or Gemma4 12B easily and i have a Pi5 8B that can run a 3B model well. I wa

Hardware

I ran the actual break-even math on buying vs renting an H200 box, and it is not where I expected

Every rent-vs-buy thread I read has confident people on both sides, but not many actually show the numbers. So I finally ran the numbers for our own decision. Posting the working here in case it is us

hardwarer-localllama

Jev vs. Kev: open-source Jev alternative tested side by side

We hosted Kev 4B (Jared Palmer's Apache-2.0 fine-tune of Qwen3.5-4B) and ran it side by side with Jev on the same endpoint to see how it compares. We built a fresh set of 362 items published after bot

Kev 4B topped out in every Tetris game I ran. Mica v0.1 4B cleared about 4x more lines and survived two of them to the end

I had Mica (my 4B decision model) and Kev 4B play the same Tetris games, same seed and same piece order, one RTX 3090. For context, this is a side project. Training and all the experiments ran on rent

LLM on a budget part 2, from P102-100 to CMP 50HX.

I finally got around to upgrade the GPU's. First a word of warning, when upgrading GPU's on P520 you have to be extra careful not to slot the card on any angle other than straight when installing or w

M5 Ultra 80Core GLM-5.3-Flash on DwarfStar Speeds

I've been playing around with various models on the M5 Ultra 256GB 80-core Mac Studio. These are the results over many rounds of agentic inferencing. I'm happy with the performance. Glad to have the l

Make Volta Fast Again

For those who have V100 cards, I wanted to point you to 1Cat-vLLM, a vLLM fork that enables optimized serving for these cards. Showing stats for Qwen3.6-35b comparing a Strix Halo with a hughly optimi

Mica v0.1 4B got an iron pickaxe in real Minecraft without generating a single token

Mica v0.1 4B playing a real Minecraft 1.20.4 server. Video attached. How it works - Each step the bot's live game state (inventory, nearby blocks, entities, last result) is written out as text. - Mica

Mica v0.1 4B: open Jev-style decision model (yes/no, choice, score) that runs on an 8 GB GPU — trained for under $30 of GPU time

I've been building a small decision model for agent loops: gates, routers, 'should I ask the user or just act' checks. It's out now as Mica v0.1 4B (Apache-2.0). What it does You give it a state, a qu

out of these two options what will give the best performance?

I am new to this and a week or so ago I setup a local ai box, it is an amd 9950x with 64gb ddr5 6000 ram, 2 rtx 5090s (on motherboard that does pcie gen 5 x8 per card). It is working fine but I have b

PSA for M5Ultra owners running LLMs: set your prefill step to 8192

Prefill step size affects both PP performance and your drafter fetching logits for MTP (affects dflash as well, mlx-vlm needs to be patched slightly to support chunked prefill for dflash). It also nee

Qwen3.8-27B: Using KV Cache Transplants to Boost Output Quality

Since my last post, I've been thinking about different options for dynamic performance degradation, trying to squeeze as much high-quality inference out of my GPU as I can. Over the weekend I read thi

Qwengram-0.8B: I transferred Qwen3.8 Flash-Next’s n-gram memory into Qwen3.5-0.8B — 5.05% lower validation perplexity

I’ve been experimenting with whether Qwen3.8-Flash-Next’s pretrained PLE n-gram memory can improve a much smaller Qwen3.5-0.8B model. I trained the 0.8B setup with limited resources, mostly using free

Splash on a 40-core M5 Max: +20% decode by tuning the kernels for your own chip

FYI the engine's default kernel rules were measured on smaller chips (16/20-core M5s and a 32-core M4 Max), so a 40-core M5 Max runs guesses. Splash's repo includes a developer tool, “make tune-kernel

Swift1.5-Qwen3.8-Flash-Next is phenomenal vs. base 3.8-Flash!

TL;DR - Swift Flash is a killer model that massively reduces excess reasoning. Try it out! If you haven't seen from my previous comparison posts, I'm a huge fan of the Swift Qwen3.8 models. I've been

Tired of RAM prices, so I have been pooling spare RAM across old devices to run bigger local models

Been dealing with the same thing a lot of people here probably are - RAM prices keep climbing and buying a single machine with enough memory for the models I actually want to run just isn't realistic

Trained my first small language model

I have a tool that uses Gemini Flash with the lowest thinking budget to do summarization work. It's very fast, 0.9-1.2s in most cases. But I have a user experience problem where people make the wrong

VLLM 4x rtx 3060 vs 8x rtx 3060 performance loss

Hello! I am currently building my local AI server, I have the Huananzhi H12D-8D EPYC Motherboard with 8x16GB memory sticks at 2666 mhz (waiting for the other components at the moment) I currently have

What IDE to use for local models

Hi people, I am looking for a lightweight IDE or plugin that won't inject large context at initiation. I tried Cline and native VS Code but they inject such heavy initial context that it fills up my g