All entries

What are some practical tasks I can assign to my local AI models?

I'm looking for more information to expand my creativity around this. I don't really have a realistic idea of what people actually do with local AI yet, I mostly just want to explore the possibilities

8 Sept 2026

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Local Ai

Which local model is actually good at knowing when to stop and ask you a question?

DGX agent

I’ve been thinking about this after using more agentic/local coding models. A lot of the newer models are surprisingly good at continuing on their own. But sometimes that seems like the problem. If a

local-air-localllama

8 Sept 2026

Which local models handle structured extraction well? I moved my agent's memory into a folder (BYO model) and want to run it on Ollama instead of Sonnet

Setup, so the question makes sense: my coding agent's memory now lives in a folder of Markdown — entities, episodes, and procedures (workflows with a per-step success/fail record). Nothing needs an ac

model-releasesr-ollama

8 Sept 2026

9 easy steps for llama.cpp, a local model, Freecad (and pi coding agent) to generate solid objects that sound mechanically good and can be also be 3D printed/milled

Quick setup on linux: STEP 0: install/download llama.cpp, Freecad, your favourite gguf model - possibly with multimedia image reading capabilities (I've used Qwen3.8-27B-UD-Q4_K_M and relative mmproj-

After over a year of my nights and weekends, the Jenny app is done!

Hi all! I just wanna say that I am tired lol. Yes, it's another harness, but I spent a lot of time and effort and have forsaken my hobbies to build the Jenny (like XJ-9) app. Jenny is a free, MIT lice

Benchmarking calories evaluation with LLMs

I wanted a quick calories counter for myself, using LLMs to evaluate the calories from pictures of meals + descriptions. I needed to pick a model so I made a quick benchmark. The setup was: - Nutritio

Can Some Generous Soul Grant Me My Wish?

Please, Oh gentle soul, please beautiful soul, please kindhearted soul, with serious hardware and time to kill, please benchmark Unsloth Qwen3.8-27B-Q8_K_XL against Qwen3.8-Next-Flash-Q4_K_XL and prov

Cybersecurity is local AI model's killer use case

This weekend I posted about the gap closing between frontier models and open source models. Well, now I'm coming with receipts. I've been running local + cloud models against real public github codeba

DeepSeek-V4-Flash-Vision-Exp is amazing at creating game worlds!

Model: DeepSeek-V4-Flash-Vision-Exp (local and API when impatient) Time: about one weekend (2 days) of QA and small improvements Full game is here After Qwen3.8-Flash-Next one-shotted a really cool Ca

Draft acceptance tweaks?

What can I do to get more from MTP if anything? It am I sitting at the expected range. Thanks /opt/llama.cpp/build/bin/llama-server --host 10.210.44.126 --port 11434 --model /opt/models/Qwen3.6-35B-A3

exllamav3 comfortably beats llama.cpp running CPU-offloaded Qwen-3.8-Flash-Next on my setup!

I got 2x 20GB RTX 3080s + 128GB of DDR4 2666hz RAM (only 4 of 6 channels populated) + a Xeon 6148 I've always been a llama.cpp person and I've been running Unsloth's Q4_K_XL quant of Qwen 3.8 Flash Ne

ExLlamaV3 is underrated

I moght get shit on for posting this but, I feel like i don't see this being talked enough and it feels like such a waste of a good piece of software. Exl3 is incredible, albeit only if you have NVIDI

gazecom : an interactive and agent-driven wrapper for comfy

Before the introduction, two caveats. This isn't anything as spectacular as the mmh3 stuff filling these subreddits these days, so please go easy on me. It's an experimental tool I'm sharing in its ra

Higher acceptance length, slower prose: Ling’s n=1/2/3 MTP test on one Spark

The missing control is visible in sudoingX’s Ling-3.0-flash benchmark graphics. The earlier table leaves Ling’s no-speculation baseline as “not measured.” The later code/prose graphic fills it in: abo

I made Warrior Quest, a local LLM-powered dark-fantasy RPG where the model only plays NPCs and the actual game state stays deterministic

The LLM is limited to NPC emulation. Game state, world logic, quests, and the authored story are handled by deterministic game systems rather than the LLM. I built it this way because I wanted the fre

local-air-localllama

Hardware

I Ran 112 MiniMax H3 Tests on an RTX 3090 — Searching for the “Golden” Settings

hardwarer-stablediffusion

I ran a 112-case MiniMax H3 quality sweep on an RTX 3090 last night trying to find the best combos . Kept the prompt, seed, reference image, RefMods, Sage setup, sampler, and scheduler stayed constant

I REALLY hope the new gemma 5 family sticks to the 'chat model first' philsophy and doesn't fall into the Qwen trap

It just seems every local 30b class model is just trying so hard to be the next Qwen that they all just kinda blend into a mass of code focused models. I really like how gemma 4 31b turned out with it

Industry

Is AI learning from the wrong conversations?

I was just checking I still had training turned off and it struck me... If users doing serious, complex, sensitive or valuable work are more likely to opt out of training, does that create a blind spo

industryr-chatgpt

Krea2 Turbo Distill 4 step LoRA - FINAL Version released today

Krea 2 Turbo — 4-Step Distillation LoRA Full details and to download - HF Repo: https://huggingface.co/lvladikov/Krea2-Turbo-Distill-4step-LoRA https://preview.redd.it/9p2rjwh3y3oh1.jpg?width=3072&for

Agents

KV cache as an agent runtime [R]

Our research team has been exploring an alternative approach to achieving interactivity and better responsiveness with LLM systems. One of the team members wrote up a post about it: https://research.y

agentsr-machinelearning

Laydee Latte - Pan Con Chocolate - Krea 2, Mimimax H3 & Mimax Music 3

local-air-stablediffusion

Over the last two weeks, I set a challenge for myself: to find out if it was possible to make a high-quality music video using a local rig and open-source AI models. So, in my free time, I’ve been exp

LayerStoRm open-source expert streaming: 1M context GLM-5.3-Flash [UD-Q4_K_XL] at 24.5 tok/s @8k on just 2× RTX 5090 + 2× RTX 5080 (186 GiB MoE on 96 GB VRAM)

LayerStoRm: GLM-5.3-Flash UD-Q4_K_XL (186 GiB) at 1M context on 2× RTX 5090 + 2× RTX 5080 (96 GB VRAM total) using RAM for the pinned experts. LayerStoRm is a (still experimental) MIT-licensed continu

local-air-localllama

Lit Review on Benchmarking LLMs Running in your phone!: MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments

Back to reading about LLMs as agents on your phone doing GUI tasks! This time I read about MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments a

Hardware

little-coder vs just Pi

Hi, I use hermes as a harness, and I am pleased with it, but sometimes Hermes's context size is a bit too much for my system, so I wanted to delegate or straight up use a lighter option for coding tas

hardwarer-localllama

LLM-guided program evolution improves 10 best-known circle-packing solutions (Packomania csqv, N=101-114) [R]

model-releasesr-machinelearning

I used an LLM to iteratively evolve an optimization algorithm rather than solve the packing directly. Starting from a simple seed solver, the LLM proposes algorithmic changes guided by a scoreboard of

MageTrail - Small Scale Full-Finetune To Show MageFlow 4B Potential As Foundational Architecture

Hi! Today I'm releasing MageTrail, a Danbooru/E621 proof of concept Full-Finetune of Microsoft's MageFlow 4B T2I model, using a diversity maximized condensed 41k images dataset (originally made by Lod

Measuring LLM performance drift: observations and methodology from 31,352 repeated benchmark measurements [D]

model-releasesr-machinelearning

One thing that has bothered me about LLM benchmarks for a while is that most of them are essentially snapshots. A model is evaluated, a score is published, and we tend to talk about that score as if i

My LLM framework. (people asked) (SAI)

DISCLAIMER I ramble a lot and am not good at focusing on documentation and or coherent thoughts. TLDR I did things I made a thing and it works for me. I will try to answer any questions. A few people

My Qwen3.8-27B task-aware quant reaches 99% of BF16 reasoning performance at 15% of the size.

TL;DR My TAK quant of Qwen 3.8 27b scored 82.81% on reasoning, comparted with 77.34% for the byte matched Unsloth UD IQ2_S and 83.59% for BF16. Edit: Some of you have tried coding with this reasoning-

New Benchmark: The Struggle Bench

How it works. The model being tested is given a server capable of running it's weights and full context. That server is placed in a median priced apartment. The AI is given a bank account with for ren

ParaRNN: Parallel Training for Non-Linear RNNs (sLSTM, RWKV-7, CfC, Titans) via Triton Newton Scans [P]

model-releasesr-machinelearning

I've built and open-sourced ParaRNN, a PyTorch/Triton library that parallelizes the training of non-linear recurrent architectures across time using Newton-Raphson iterations mapped to associative pre

Qwen 3.8 27B run just 0.7tok/s on 5060ti 16GB

I just tried to run Qwen 3.8 27B 4bit qwant on 5060ti 16GB and got terrible results. With Qwen 3.6 I got something around 15-20 tok/s. Are there any combination of parameters or tweeks how to push the

model-releasesr-ollama

Qwen 3.8 Next Flash is really really REALLY verbose..

Long time user of 3.6 27b, switched over to Next Flash since it's a logical step up even from 3.8 27b. It's soooo verbose, i'm talking 13 minutes of thinking time on single turn coding requests at app

Qwen3.5 0.8B on CPU

Since the Qwen3.5 0.8B model is an interesting one for small specialized fine tunes, I was curious how fast it can run on CPUs. Why CPUs? Mainly because I want to use it as a local dictation cleanup m

Qwen3.8-27B on M1 Max 32GB: MLX 15.8 tok/s vs llama.cpp 9.7 tok/s - but llama.cpp prefill is faster

I’ve been setting up an M1 Max Mac Studio (24-core GPU, 32GB unified memory) as a local LLM server and wanted to compare MLX vs llama.cpp on Qwen3.8-27B. I tried to keep the model footprint and benchm

model-releasesr-ollama

Radeon RX 7900 GRE 16GB and RX 480 8GB Vulkan benchmarks llama.cpp

My XFX Radeon RX 7900 GRE 16GB Vram GPU struggles with models over 20B size. I added my Radeon RX 480 8GB Vram GPU to the system and ran a few benchmarks using llama.cpp Ubuntu Vulkan build 10453. htt

Research

Roboticists working in Learning-from-Demonstrations and Behavioral Cloning : What is going on in your field these days? [D]

researchr-machinelearning

Is LfD and BC research being effected by recent advances in (so-called) Frontier LLMs? Or is research in LfD and BC sort of going along in an independent direction from these? Are you seeing any use f

tencent/EVIE-8B and EVIE-4.5B (High-Capacity Visual Document Retrieval)

https://preview.redd.it/3p6234jzk2oh1.png?width=900&format=png&auto=webp&s=b9eab3351d6c6dbd4d5b3fd677a6c40b57f18167 https://huggingface.co/tencent/EVIE-8B 🌟 Highlights SOTA Retrieval Performance: 66.7

The models are fine, our toolings and methods are shit.

I've hit again a point where me as a developer have to take a break from all this slop shit. Im a Developer for 13+ years and i loved it. But i fell for the slop trap. First it started with copilot an

local-air-localllama

Thinking about grabbing an RTX 2000 Ada 16gb to add to my gaming pc for inference due to Wattage constraints, any advice?

So, I’m currently just short of being able to run the mainline dense SLMs (Qwen 27b/Gemma 31b), running a i9-14900F/RTX4070 Super/32GB DDR5 setup w/ a 700W 80 Gold PSU and a MSI PRO B760-VC WI-FI IV m

This week, three hikers were rescued from a mountain after following AI's advice. On the same day, OpenAI announced that we have reached the era of AGI. The two stories together got me thinking.

On September 1st, three young men from California used Gemini to plan a hiking route on Mount Shasta. The AI told them to bring far less water and food than was actually needed for a hike of that di

model-releasesr-chatgpt

VDN-H3 on M5 Pro 24GB: up to ~2.6× faster H3 generation

I recently ported VDN-H3 (Video DeltaNet for MiniMax H3) to Vpipe and tested it on an M5 Pro 24GB. The benefit scales with both sequence length and resolution: at 832×480, VDN-H3 starts to pull ahead

Industry

Who’s still using prompts?

I keep seeing these massive ultimate chatgpt prompt posts all over social media, like we’re still using Chat and other LLMs the same way we did when they first came out. Meanwhile the more I use AI, t

industryr-chatgpt

2x R9700, 64 GB DDR5 is an absolute beast machine with vLLM Radiance / R9V and Qwen 3.8 27b and Flash next

I've been tinkering with local LLMs since the beginning of the year when I had an Intel Arc B580 and 32 GB of DDR5. Curiosity got the best of me and I bought the first R9700 about half a year ago, als

48 tg/s 440 prefill on my grandma's cluster (2xP40) (sort of)

TL;DR: switching KV cache to f16 may give a boost in speed if using MTP and ngrams. I have a self-built 'AI mega-cluster' with 2x P40s on a cheap Chinese motherboard and a Xeon CPU (around $1,100 to b

4xRadeon AI Pro R9700 people, how are your benchmarks?

Was considering choosing a 4xRadeon AI Pro R9700 build for a office AI server. Curious to know what results you guys are getting? Mainly considering DeepSeek V4 flash and Qwen 3.8 Flash with offloadin

8 uncensored Qwen 3.8 27B variants, one base, 167 GPU hours - Abliterlitics

This comparison was requested by a few people, and certainly we were all eager to see the final results. The comparison had taken 11 days and the GPU was crunching numbers for ~167 hours. We've been c

Best model + setup for remote deployment.

Hi everybody, I am in a tricky situation. My grandfather + grandmother live in a remote village in asia, and were visiting us in the city. My grandfather has absolutely fallen in love with AI, and it