All entries

DeepSeek-V4-Flash-0731: When Low is higher than High

I decided to test a few questions against DeepSeek-V4-Flash-0731. Locally, I was running Unsloth's UD-Q2_K_XL quant. After I saw the surprising shape of the results, I tested against DeepSeek's offici

DeepSeek-V4-Flash 284B on 5.3GB of memory

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

DGX agent

Following up on my Qwen 3.6 port, I wanted to keep adding models and ended up fixing a bunch of things along the way, so it's its own engine now: Mference. Same core idea from TurboFieldfare, MoE mode

model-releasesr-localllama

2 Aug 2026

DSpark Benchmark Result on Deepseek v4 Flash 0731

TensorSharp supports DSpark on Deepseek v4 Flash 0731 now. Here is the benchmark result on 4x Nvidia A40 GPUs, cuda 12.8 with/without DSpark: Model: DeepSeek-V4-Flash-0731-UD-Q8_K_XL from https://hugg

Encrypted Clouds?

I love the progress happening on open models but I feel like it is kind of getting clear that hardware to run good sized models is completely unaffordable for me right now. I know that you all love Qw

Expert-only IQ3 requant of DeepSeek-V4-Flash-0731: better KLD than UD-IQ3_S, 1.4x decode on a CPU-spill rig

Hey all, tldr / who this helps: you run a mixed multi-GPU box where the experts spill to RAM, and you want to stay in the 3-bit tier instead of dropping to Q2 to make it fit. https://huggingface.co/Ta

Got '403 Your account is currently unavailable' on Ollama Max after 2 weeks — No reply from support in 3 days. Anyone else?

Hey everyone, I'm reaching out to see if anyone else has run into a similar issue or if there's an alternative way to reach their team. I subscribed to the Ollama Max plan on June 17th. Everything was

Agents

How do you test your setup?

We all have been there, tinkering around with models is fun but we rarely do it with research precision and issues are often subtle and hard to reproduce. There are a lot of benchmarks but running the

agentsr-localllama

How well do multiple GPUs scale for LLM inference? (Trying to understand the basics)

Hi everyone, I’m fairly new to the multi-GPU side of local LLMs and I’m trying to understand how inference actually scales across multiple GPUs. Suppose I have a model running on a single GPU and then

I built a self-hosted studio that turns one reference photo into a curated, captioned, trained and tested LoRA — one browser tab, open source, MIT

local-air-stablediffusion

I shared this tool here a week ago and the feedback shaped a big new version, so here's the full tour of what it does today. Screenshots of every screen: github.com/perfectgf/lora-dataset-studio — plu

I built an open-source LLM Gateway to route and fallback between local Ollama models and cloud APIs

Hey r/ollama 👋 If you run Ollama locally alongside cloud endpoints for agent workflows, Cursor/Windsurf, or custom scripts, managing API switching, failover logic, and context limits can get messy fas

model-releasesr-ollama

Hardware

I pushed Kimi K3 onto one CPU with 8 GB of RAM

I deployed K3 on 32 H100s at work a couple of weeks ago and then got annoyed that there was no way to poke at it on my own machine. So I wrote an inference engine for it in C99. Nothing clever going o

hardwarer-localllama

Is training a WAN Lora going to fix the identity shift in I2V?

local-air-stablediffusion

Using a character Lora image from Krea2 as my first frame but even for a subtle camera move there is identity loss immediately. Will training a character Lora for Wan2.2 help fix it? I've never traine

It looks like MiniMax H3 uses Qwen3-VL-32B as Text Encoder and has has a split Transformer

local-air-stablediffusion

I'm trying to get more hints from this PR in comfyui github https://github.com/Comfy-Org/ComfyUI/pull/15210 but from what I've gathered so far it looks: It uses Qwen3-VL-32B as the encoder (50 layers

MiniMax H3 is going open-weight in under 6 hours

model-releasesr-stablediffusion

here is all the info we have based on open PRs to add support to ComfyUI and HuggingFace diffusers - 33B for the main DiT and a pruned 20b variant - Qwen-3-VL-32b as the text encoder Edit- I posted cl

Parlor v2: best-effort fully local GPT-Live clone on an M3 Pro

GPT-Live is so good that I use it almost every day. I've been wanting to replicate it since it was released. My first attempt was to fine-tune Gemma 4 12B to behave like a full-duplex model. Something

PSA for DeepSeek-V4-Flash-0731 users — don't blow out your prompt cache with system role messages mid-conversation

DSv4F doesn't ship a jinja, but for distributions that do and faithfully reconstruct what DS releases in their chat template python, every system message is hoisted into the system prompt at the top -

PSA: llama.app, Mac app and llama serve from llama.cpp

https://llama.app/ Been using llama.cpp for years now and im on here all the time (im a mod..), but somehow I totally missed that llama.app exists and its official from the HF/llama.cpp team. So posti

Ran DS V4-Flash-0731 Locally on 3xMI50 32GB @ ~15 t/s TG

Hey y'all. I'll be concise. TL;DR: DS V4-Flash-0731 @ UD-IQ2_M running fully in VRAM on 3xMI50s (90.9 GB model, 96 GB VRAM). Actual speed on llama-server is: - Text Generation: ~15-16 tokens/second st

Real-world reality check on Qwen for autonomous coding agents

TLDR below 👇🏼 I’ve seen a lot of hype around Qwen 3.6 35B and 3.5 120B lately, especially regarding coding and tool-use capabilities. On this subreddit it is the defacto recommended model for everyone

[Release] WinterMix — Qwen3.5-122B-A10B in native MLX: an 82 GiB build that beats 94–95 GiB quants, plus a 68 GiB build for agent swarms

TL;DR: I spent 9 days developing a new quantization method for MLX models and measured 18 variants against each other on a single M5 Max MacBook Pro (128 GB). The result is the best-measuring MLX quan

Released a Windows Agent Server for Reins/Ollama with Self-Healing Execution Loop (Standalone .EXE included)

Hey everyone, I’ve built a Windows Middleware Agent Server designed to pair local LLMs (via Ollama) with frontends like Reins App. Key Features: Self-Healing Loop: If a generated PowerShell command fa

model-releasesr-ollama

Running DeepSeek-V4-Flash-0731 (155 GB MoE) on a DGX Spark with vLLM-Moet 2-bit quantization - AI's narrative

# Running DeepSeek-V4-Flash-0731 (155 GB MoE) on a DGX Spark with vLLM-Moet 2-bit quantization I used Deepseek-v4-Flash-0731 cloud API settig up vllm-moet to run deepseek-v4-flash with MTP locally on

Single system with dual cards or two systems with single cards?

So I am in a conundrum and I'm thinking of asking for your opinion for the following: Currently, I have a 5800X3D gaming rig with a 7900XTX with its 24GB VRAM. It seems that for this subreddit, this c

Try handling complex tasks to your local models with GraphARC, graph engineering yes !

🚀 We just built our first real-time implementation of Graph Engineering, inspired by our experience building graph tooling used by 4,000+ developers. 🔗 Repo: https://github.com/CodeGraphContext/grapha

local-air-localllama

Trying to setup two LLM’s to run on 2 gpus separately but simultaneously on one machine.

Okay so this probably sounds like kind of a dumb setup but hear me out. I have a 4070 I’ve been running gemma4 off fine but I recently slapped in a spare 1650 I’ve had laying around to run a second li

Vacuum 16T

https://huggingface.co/tsfrm/vacuum-16t A 16.5-trillion-parameter model that contains nothing. This model is just a ████ you to the labs and companies who say that 'haha I have the biggest model out t

What’s the community’s favorite benchmark to validate performance?

Built my 1st inference machine and have been tweaking models trying to get the most out of my modest hardware. I think I’m at a good place but I’m testing with my own prompts. I’ve looked into some of

When did Ollama become so cool with non self hosted models?

It feels like Ollama’s brand identity has shifted. It used to be centered on self‑hosted models, but now most of the new releases seem to be API‑based services with far more emphasis on cloud integrat

Why are almost all new benchmarks and leaderboards coding focused?

I know in in this community LLM's are generally used for coding but there are other usecases besides coding and those usecases should be tested too. I also know benchmarks can sometimes be benchmaxxed

Xberg v1 is out

Hi all, I'm happy to announce that Xberg v1 is out. Xberg is the successor to Kreuzberg, equivalent to what would have been Kreuzberg v5. It's a content intelligence framework that handles a very wide

You really should not quantize KV Cache for DeepSeek V4 Flash

I don't think anyone should quantize the KV with DS4F. I checked the the quality impact (PPL, KLD, Same TopP) for swhitching from BF16 KV to Q8 KV, and it appears significant. Very much in contrast to

A collection of small domain-specific benchmarks for local models (30+ and growing)

Hello fellow local AI people! I took 'you must create your own benchmarks' literally, and built a website for this. How does the end result look like Let's say I want to know which model has most comm

Are 1B LLMs Going Away in 2026?

I don't know much about llms aside from downloading them through a frontend and running them on my laptop or potato phone. Google released gemma 4, but unlike gemma 3, there isn't a 1b model this time

[audio.cpp] Release 0.5: DramaBox expressive TTS, Confucius4 cross-lingual voice transfer, plus 7 more models and ROCm/HIP

audio.cpp 0.5 is out :) The most fun new model in 0.5 is DramaBox. It is closer to prompt-directed voice acting. DramaBox is built on the LTX-2.3 audio architecture, and prompts can control emotion, d

Best free tier cloud models?

I'm currently using the gemma4:31b-cloud, and it is pretty good, but sometimes it gets confused. Are there more free tier models out there I should try out? or is gemma4 the cap of free tier cloud mod

DeepSeek V4 Flash 0731 IQ2_M benchmark for Dual 3060 and 96GB RAM ≈ 3.5 tok/s.

Thanks to the community help I finally launched this llm. LM Studio refused to load weight onto second GPU but Unsloth Studio did so everything was done in there. Not a proper benchmark (used PC in pa

DeepSeek-V4-Flash-0731: Models you can run locally now have the intelligence score of the top frontier model from March 2026

March 6th, 2026 the highest intelligence index score was 51 for frontier models. deepseek-ai/DeepSeek-V4-Flash-0731 that has an intelligence score of 50. If these benchmarks are accurate, models avail

DeepSeek-V4-Flash-0731 on Bosgame M5 with RTX PRO 6000 Max-Q eGPU

Here are my numbers: Quant Size Layout Decode Prefill Draft acceptance UD-Q8_K_XL 150.8 GiB 20 layers CUDA0 / 23 ROCm0 + drafter 44.0 t/s 564 t/s 0.535 UD-Q4_K_XL 144.4 GiB 22 / 21 + drafter 48.4 t/s

Deepseek v4 flash 0731 still not holding up.

The biggest issue with preview was its inability to follow rules prompts and skills. It seems like no matter what you do it ignores them. I've tried first person and second person. I've tried Chinese

DeepSeek-V4-Flash-0731 UD-IQ3_S 12.5 tok/s on RTX 3090 +128GB DDR5

I managed to run DeepSeek-V4-Flash-0731 UD-IQ3_S in text-generation-webui with: RTX 3090 24 GB 128 GB DDR5 overclocked to 5600 MHz using AMD EXPO llama.cpp loader First, I had to use a rather brutal w

DeepSeek-V4-Flash-0731-UD-Q3_K_XL 3x3090 test results

For anyone interested, here are the llama-bench results on 3 bit K_XL quantization. I think this could be pushed further but no luck so far. CURRENT RESULTS: full moe offloading Prefill suffers 116 --

DeepSeek-V4-Flash-Q4KExperts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2-imatrix-0731.gguf

Antirez stealthily uploaded the new weights in the old folder... and there we were tapping our fingers. https://huggingface.co/antirez/deepseek-v4-gguf/tree/main submitted by /u/challis88ocarina [link

DS4 flash 0731 - Acquarium Panel Failure - Q3_K_XL Unsloth

https://preview.redd.it/1a39x4zivqgh1.png?width=1550&format=png&auto=webp&s=de591c039cc18782a6b5d8e402fdc1594be05132 start C:llmllamam5uildinllama-server.exe --model 'H:UD-Q3_K_XLDeepSeek-V4-Flash-073

For MoE models the arithmetic splits in two: capacity follows total params, speed follows active

A few people asked for this after the bandwidth thread, so here it is on its own instead of buried in a comment. The dense rule was simple: every token reads every weight, so tokens/sec ≈ bandwidth ÷

Safety

Github repo to learn the OPD/OPSD and how they perform compared to GRPO, on a consumer grade GPU [P]

I am trying to learn concepts like On Policy Distillation (OPD), On Policy Self Distillation (OPSD) and how do they compare to RL algorithms like GRPO. There are a lot of papers on this, but because o

safetyr-machinelearning

Huawei 96gb

Ciao a tutti ho appena comprato 4 Huawei duo 96 GB a poco più di 4mila euro qualcuno ha già utilizzato CANN consigli da darmi? E la prima volta che utilizzo questo framework e non so proprio da dove i

I built Vao2, an open-source personal feed for News, YouTube, GitHub and more with local AI summaries

GitHub: https://github.com/Loann110/Vao2 Hello everyone, I’m currently developing Vao2, an open-source application that brings together news, YouTube channels, GitHub repositories and more (to be adde

Is there a point where models just cannot get any smaller without losing intelligence?

DeepSeek V4 Flash got me thinking... We keep seeing smaller models get way better. A model at a certain parameter count today can be much smarter than a model of the same size from a year or two ago.