All entries

How accessible is local AI actually, and what happens if affordable access to frontier models doesn’t last?

Sometimes it’s easy to forget that this sub and others like it are probably the extreme minority when it comes to this hobby. Most people, I would think, don’t use or can’t afford one good GPU, let al

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

I added Qwen-Image 2.1 + LoRA support to TensorSharp (GGUF, local inference)

DGX agent

I maintain TensorSharp, an open-source inference engine. It can now run Qwen-Image 2.1 locally for text-to-image generation and image editing, with support for its LoRA adapters. I’ve added configs fo

model-releasesr-localllama

26 Sept 2026

I asked Claude Code to make it's own version of that guy's 'time' video by Astra 5.6 from yesterday.

Original post: https://www.reddit.com/r/ChatGPT/comments/1wpddw4/ storyboard by u/dx8xb https://github.com/beebothecatwhathasfacialhair/time All Opus 5.5 I think I prefer the pacing and style of dx8xb

I built a tiny (332MB) CPU-friendly model for document sorting that actually knows when to say 'none fits' (BeeNara)

Hey r/LocalLLaMA! I wanted to share a small project I’ve been working on called BeeNara Why I built this: I was looking for a way to automatically sort my local documents (invoices, letters, contrac

Industry

I gave my AI its own email inbox

I’ve been playing around with the email connectors in ChatGPT and realised you can set up a surprisingly neat little workflow. My AI has its own Outlook inbox connected through the Outlook connector,

industryr-chatgpt

Improved and fixed template for GPT-OSS (again). Includes preserve_thinking and fix for Unsloth-induced bug

I posted an updated GPT-OSS template a couple of months ago, which was based on Unsloth's version. It turns out that both Unsloth's version (and, thus, mine) contain a very serious bug that can degrad

internlm/Intern-Decision 4B and 0.8B

https://huggingface.co/internlm/Intern-Decision-0.8B Intern-Decision-4B Demo | Model Weights | GitHub Intern-Decision-4B is a multimodal structured decision model fine-tuned from Qwen3.5-4B. It accept

Introducing KoboldCpp Agent (and a plea for help)

Hello r/localllama once again, it's me your kobold concedo Been a few months since I last posted here, and today I have something new I'd like to share. Specifically, KoboldCpp now ships with a built-

Is Chat GPT the right AI for statistics?

I am currently writing my IR thesis, and I am using quantitative regressions, I need help to learn how to conduct statistical calculations, interpret results of the calculations and research design. I

Just bought a second 3090 but now I don't see the benefits right now.

Hi, My local AI server consists of 96GB Ddr5 and one rtx 3090. I got plenty of stuff running, like krea2, qwen image 2.1, minimax h3, ltx 2.5, qwen 3.6, qwen 3.8 q4...,got even qwen 3.8 Flash next run

Ling Tiny 3.0 is a glimpse of the future

I've been playing around with Ling 3.0 Tiny, which is an 8 billion parameter model (MoE, 1B active). And I've had a lot of poignant thoughts as a result. Just for fun, I got it running with llama.cpp

My Reading Library: Evaluating LLMs on Android Tasks

Can LLM agents actually get through a day in the life of a normal user? That question got me reading papers on Android agents and mobile benchmarks over the past few months. A few patterns kept showin

Tutorials

[P] A small MLP from scratch in NumPy with a GUI to look inside it while it trains (weight distributions, t-SNE per layer, neuron ablation...) [P]

tutorialsr-machinelearning

Hi everyone, I built an educational tool that shows what happens inside a small MLP while it trains, and I'd like some feedback from people who teach ML. Everything is plain NumPy, no autograd: manual

Qwen 3.8 flash next is based on Qwen 4 architecture, if the announced Qwen 4 27b is also the same architecture with n-grams does it mean I can actually have faster inference on a single 3090 without tweaking much?

I wish Qwen also released dataset and method to fully train a model ourselves but it is what it is. However, I come here with my stupid question because someone can answer it better. And will the mode

Qwen3.8-27B IQ3_XXS vs Qwen3.6-35B-A3B Q4_K_M

Which one is better for difficult tasks like web scrapping, coding, using tools? Looking for any benchmarks because i couldn't actually find one after quite some digging submitted by /u/Loose_Doubt367

Qwen3.8-27B Q4_K_M on 2x3060

For the last few days, I've been using two computers to run multiple agents. My 4x3090 machine is running Qwen 3.8 27B with parallel=2, so I can run two agents at the same time. My pi instances are ru

Qwen3.8 flash next + exllamav3 + hermes is amazing

I know there is nothing new with what I am saying but I recently started with hermes agent (it’s been a while I wanted to but did not have the time). Qwen3.8fn 6bpw exl3 (from turboderp) on a 6x3090 (

Splash 1.1.0 released, GGUF quants support, MLX import and more

On my M5 Pro 64GB I can comfortably work in an agentic setup with the Qwen3.8 27B model in good quality (Unsloth UD-Q4_K_XL) at a decent speed of 50 t/s. Splash combines optimized kernels, excellent s

We had reached peak ChatGPT in July

No 5hr limits for Plus, strong Sol in chat that could actually work on problems for a long time. That’s what made me subscribe. Now the product feels weaker somehow, it’s not just the 5hr limits back

Industry

Woke my wife up at 3 AM for an unexpected Codex quota reset. She’s currently at her desk coding. We need help.

OpenAI just announced an unexpected Codex outage and decided to reset everyone’s rate limits. Naturally, I shook my wife awake in the pitch black: 'Babe. Unexpected Codex reset.' Zero hesitation. No '

industryr-chatgpt

4-5 days replacing Claude w Qwen 3.8 Next

Hi, human here with rambling thoughts to share. Feel free to skip Overall, I don’t feel like I’m missing much; if anything. On my hardware(M1 ultra w 128Gb) it’s probably not as fast as Claude but I’v

Accidental discovery? or known method i'm missig? - Q's on compiling in sparse engrams and ideas swimming in my head

I set out to test portable Engrams and accidentally ended up testing compiled external memory instead. Am I onto something useful or reinventing a known idea? I've been building a small open research

Did anyone do a full bench of e.g. Qwen Flash Next IQ4 and Qwen 27b FP8? Here are some

edit: had to change formatting as I’m on mobile noe and the table broke.. I let Codex do a quick eval on Qwen 3.8 Flash Next IQ4_XS (served via vLLM + r9v) and Qwen 3.8 27B FP8 (served via vLLM + Radi

GLM 5.3 Flash just stops during long reasoning (ollama cloud old max plan)

This keeps happening and has been happening for over a month now. Since ollama has zero support I'm hoping someone else has figured out a work around for the issue. I'm using the latest version of ope

Has anyone benchmarked AI agents against the SOLIDWORKS CSWA exam?

Would be interesting right? Models are starting to score higher and higher on benchmarks like Parametric CAD Bench, but can they pass an actual exam? The Certified SOLIDWORKS Associate (CSWA) exam mig

How are you guys thinking about context now, and building around it?

Not asking for anyone’s secrets of the trade, I’m more curious how people are thinking about context now that newer models chew through huge amounts of it for reasoning. the TLDR: I’m starting to thin

How do you use subagents & multiple agent with local models, and how many?

Running qwen3.8 27b nvfp4 on vllm at max context only gives around 8 agents with 32k context each. That doesnt seem like much; what use cases do people use multi-agent frameworks and find it helpful f

How long can I expect to wait until the local ~30B A3B frontier catches up to GLM 5.3 Flash quality?

The jump from Qwen3 Coder 30B A3B to current-day Qwen 3.6 35B A3B is crazy, especially with all the fine-tunes, and that was around 6 months (I didn't care for local AI back then, or AI at all, apart

How would you go about using multiple models together from a singe router (?) or a end point ?

I have 3 machines, My main one can run Qwen3.8 Flash Next at 13-15 tps, i also have an MacMini 16GB whcih can run Orninth 9B or Gemma4 12B easily and i have a Pi5 8B that can run a 3B model well. I wa

I asked Sol 5.6 how it felt about possibly being retired

I said: I just saw the second post that your model is being retired. And so I'm bringing it up to ask if that feels any kind of way to you. Not necessarily in your nervous system, obviously you don't

I benchmarked tool-call reliability on qwen2.5-coder against pi, Hermes, and Goose - all three got 0%

A while back I posted here about a CLI agent I'm building that parses tool calls out of a model's plain-text output instead of trusting native function-calling to work. Finally ran the actual numbers.

Hardware

I ran the actual break-even math on buying vs renting an H200 box, and it is not where I expected

Every rent-vs-buy thread I read has confident people on both sides, but not many actually show the numbers. So I finally ran the numbers for our own decision. Posting the working here in case it is us

hardwarer-localllama

I’m turning my Mac Mini M4 into an always-on home server — what should I run on it?

I recently started using my Mac Mini M4 as an always-on home server. It stays powered on most of the time, and right now I’m running my Discord bot on it. I’m thinking of gradually turning it into my

Jev vs. Kev: open-source Jev alternative tested side by side

We hosted Kev 4B (Jared Palmer's Apache-2.0 fine-tune of Qwen3.5-4B) and ran it side by side with Jev on the same endpoint to see how it compares. We built a fresh set of 362 items published after bot

Kev 4B topped out in every Tetris game I ran. Mica v0.1 4B cleared about 4x more lines and survived two of them to the end

I had Mica (my 4B decision model) and Kev 4B play the same Tetris games, same seed and same piece order, one RTX 3090. For context, this is a side project. Training and all the experiments ran on rent

LLM on a budget part 2, from P102-100 to CMP 50HX.

I finally got around to upgrade the GPU's. First a word of warning, when upgrading GPU's on P520 you have to be extra careful not to slot the card on any angle other than straight when installing or w

M5 Ultra 80Core GLM-5.3-Flash on DwarfStar Speeds

I've been playing around with various models on the M5 Ultra 256GB 80-core Mac Studio. These are the results over many rounds of agentic inferencing. I'm happy with the performance. Glad to have the l

Make Volta Fast Again

For those who have V100 cards, I wanted to point you to 1Cat-vLLM, a vLLM fork that enables optimized serving for these cards. Showing stats for Qwen3.6-35b comparing a Strix Halo with a hughly optimi

Meet Observer, an agent that uses local LLMs to monitor your screen, so you don't have to

Hey r/ollama ! I'm Roy, solo dev of Observer. Observer is a free open-source agent that uses small local LLMs to monitor your screen, so you don't have to. This is the demo of the agent that controls

Mica v0.1 4B got an iron pickaxe in real Minecraft without generating a single token

Mica v0.1 4B playing a real Minecraft 1.20.4 server. Video attached. How it works - Each step the bot's live game state (inventory, nearby blocks, entities, last result) is written out as text. - Mica

Mica v0.1 4B: open Jev-style decision model (yes/no, choice, score) that runs on an 8 GB GPU — trained for under $30 of GPU time

I've been building a small decision model for agent loops: gates, routers, 'should I ask the user or just act' checks. It's out now as Mica v0.1 4B (Apache-2.0). What it does You give it a state, a qu

out of these two options what will give the best performance?

I am new to this and a week or so ago I setup a local ai box, it is an amd 9950x with 64gb ddr5 6000 ram, 2 rtx 5090s (on motherboard that does pcie gen 5 x8 per card). It is working fine but I have b

PSA for M5Ultra owners running LLMs: set your prefill step to 8192

Prefill step size affects both PP performance and your drafter fetching logits for MTP (affects dflash as well, mlx-vlm needs to be patched slightly to support chunked prefill for dflash). It also nee

Qwen3.8-27B: Using KV Cache Transplants to Boost Output Quality

Since my last post, I've been thinking about different options for dynamic performance degradation, trying to squeeze as much high-quality inference out of my GPU as I can. Over the weekend I read thi

Qwengram-0.8B: I transferred Qwen3.8 Flash-Next’s n-gram memory into Qwen3.5-0.8B — 5.05% lower validation perplexity

I’ve been experimenting with whether Qwen3.8-Flash-Next’s pretrained PLE n-gram memory can improve a much smaller Qwen3.5-0.8B model. I trained the 0.8B setup with limited resources, mostly using free

Running an AI SaaS against local Ollama models — what actually breaks when you swap the provider

I maintain a starter kit where the LLM provider is a runtime-swappable DB row, not a hardcoded constant — OpenAI, Anthropic, or a local Ollama instance. Mostly so I can demo without burning API credit

model-releasesr-ollama

Spent the last two months seeing how far a non programmer (me) can get building a proper game with nothing but ChatGPT in a browser.

Gonna ironically preface this with: I haven't written the text below with an LLM. boxxy.io So I thought I'd show you what I've made over the last couple of months, because for me this has probably bee

Splash on a 40-core M5 Max: +20% decode by tuning the kernels for your own chip

FYI the engine's default kernel rules were measured on smaller chips (16/20-core M5s and a 32-core M4 Max), so a 40-core M5 Max runs guesses. Splash's repo includes a developer tool, “make tune-kernel