All entries

A fresh llama.cpp PR (#26689) changes what looks like a tiny SYCL FlashAttention dispatch decision. With a quantized KV cache ('q4_0' / 'q8_0'), decode was being sent through the VEC kernel. On the au

I swear AA is not the bipartisan they so claim. An open source mode (Qwen 3.8 max) was number 1 on the agentic index, then they just so happen to launch 'v4.1.1' of their index in which they just adju

Hi there! I've messaged the Ollama support team 4 times with no response in 3 weeks. This is getting ridiculous. Does anyone have any recommendations as to how I should seek support? I don't want to i

High-performance inference of NVIDIA's Parakeet TDT 0.6B V2 English transcription model, in the browser. Check out the live demo: https://parakeet.narcotic.sh/ A fully custom, dependancy-free implemen

I'm considering replacing a single RTX 3090 with two ASRock AMD Pro R9700s for about 2900 new out of the door. That would move me from 24GB to 64GB VRAM. Yes yes, CUDA/ROCm, but the real problem is po

I run the following on a 5090 and have been okay with its performance, it does most things somewhere 80-100 t/s, though that can slow down at full 262k context - more like 40 t/s at times. I use it pr

Hey all, I'm serving DSv4Flash 0731 on a cluster of 2x DGX Sparks but am running into constant issues with having almost no RAM (unified memory) left for the OS/cache and I'd love to hear the communit

📝 Introduction We present Wan-Animate-2, a novel end-to-end character animation framework that directly consumes driving videos in a redesigned Diffusion Transformer, which achieves high-fidelity moti

For a long time now, the most popular posts on LocalLLaMA have been either about using LLM in the cloud or about politics. I suspect that people using local models are about 10% now. You can say that

Following up on yesterday's post about running everyone's faves on 2 x 16gb cards while maximizing performance and KV. Previous post data used abandoned Cu130 VLLM image. Stats here are done on cu129-

https://preview.redd.it/kihat320ashh1.png?width=1672&format=png&auto=webp&s=a7ccc40ba3fb229ac7ebf57e8e6a314e0ee45646 Hi r/StableDiffusion! u/New-Requirement1419 -> dacongya (Head of H3 Researcher) u/A

A small update to Sir Shortoken. Sir Shortoken already had Quick, Balanced, Deep, Bullets, and Aggressive Bullets. I wanted something between Bullets and normal prose. So I added LELP-S+ (Less English

TL;DR: On a Qwen3.6-35B-A3B Q6 setup sized for 64K context on a 24GB RTX 3090, spilling eight MoE expert layers to CPU freed enough VRAM to increase -b from 512 to 1024 and -ub from 128 to 512. Prompt

Hi all. These are my system specs: dual xeon e5 2696 v2 , 160gb DDR3 ram ECC(1600mhz), 3 gpus: 3060 12gb, p100 16gb, 3050 6gb. And a 400gb nvme sdd RAID0, 3000 mb/s. The model is Deepseek-flash-0731 U

Looking for best current solutions for combining cloud models and local models seamlessly inside a harness' orchestration Edit: Right now, we don't have harnesses (that I'm aware of) that are blending

Up until now, I’d always assumed it was basically just transcribing what I said, feeding the text into the model, and reading the response back. So out of curiosity, I asked if it could actually tell

First of all, my setup: Ryzen 9 5950x DDR4 3200Mhz 64gb (2x32) Dual 3090s, no NVLINK Runtime: llama.cpp Nvidia Drivers 610 Windows 11 25H2 Qwen 3.6 27B Q8 I've been using llama-server with --split-mod

J'ai consacré beaucoup de temps à l'optimisation de DeepSeek-V4-Flash-0731 GGUF sur une seule RTX 3090. Mon exigence absolue pour chaque configuration était la suivante : Le modèle doit rester utilisa

So I can either pull the trigger on a 128gb AI max+ 395 laptop or wait for RTX Spark for LLMs. Maybe I get it now and the price of the spark is super high so it's a good purchase or maybe the Spark sh

https://preview.redd.it/o6ik6qboeohh1.png?width=1134&format=png&auto=webp&s=4016f26c50c1d93bd3d0c7e880e9b55a2d75310f I have been running Qwen3.6 27b for a little while (mostly coding tasks) and recent

Just came across this coding benchmark: SciCode Artificialanalysis.ai reports a ranking which contradicts the feeling we've towards those models in real life coding. Is Gemma 4 really that good, or a

I cancelled my pro plan ealier because I wanted to use new Deepseek v4 flash 0731 which was available on Openrouter through API only (not yet on ollama cloud at the time). The old Deepseek v4 flash/pr

As for me, I own a system with an RTX 5090, Ryzen 9 9950X3D2, and 64 GB of DDR5. Every time I see research come out with a new way to train AI, I immediately think to try it on my system to see the re

Seen a ton of posts today about the DeepSeek API price hike. Half the feed is doom-posting, the other half is explaining basic GPU economics. Honestly, I get the cost side. Sub-cent tokens were never

[Fully open source under GPL3, made from the ground up for use with local models, no subscriptions, no corporate backing] When i first started this, it was meant to be a fully lightweight, extremely m

I'm the author, so discount the enthusiasm accordingly. This is an unaffiliated community port, not endorsed by the vLLM project, which it uses to verify its correctness. What started it: I love vLLM,

I built this because the existing benchmarks were using random data and with MTP content types can vary a lot on what performance you see. 5% or more with content types. BetterBench is designed to hav

Link to the article: KV Cache Quantization Benchmarks: KVarN, Precision Tail KLD benchmarks with BeeLlama.cpp v0.4.0, fork of llama.cpp with more KV cache quantization options. Models: Qwen 3.6 27B Q5

Salve a tutti. Sto valutando di mettere dei modelli locali, magari su LM studio o altro software se mi spiegate il perché da utilizzare sia come PT sia per Soc L2/L3. Ho un PC con 128 GB RAM ddr5 8gb

NVIDIA Nemotron Parse 2.0 transforms document images into structured, machine-readable representations with text, layout classes, bounding boxes, and reading-order information. Given a Red, Green, Blu

nvidias nemotron omni is open weights and it sees, hears and reasons. theres already a 4bit mlx quant on hugging face but only the text backbone loads with standard mlx tooling. the model card says it

🐦‍⬛ Magpie-TTS Multilingual 🦜 Nemotron Speech Streaming EN 0.6B 🦜 Nemotron-3.5 ASR Streaming 🦜 Parakeet CTC 1.1B 🦜 Parakeet TDT 0.6B v3 🥦 NanoCodec Merged PR https://huggingface.co/nvidia/magpie_tts_m

GGUFs here: https://huggingface.co/ReadyArt/gemma-4-31B-it-scotoma-2-GGUF Disclaimer: By slop, we are specifically talking about specific tics with the model(sentence structures), but this doesn't inc

I love to see these impressive models coming out that compete with the giants from companies like Z.ai, Moonshot, Alibaba, etc. A win for the open source/weight community is always welcome. While I am

Users also report that the free version was significantly downgraded after the release of the new models this is very important for us when considering local hosting. A lot of people decided not to bu

So I had been building ScreenMind, kinda like local ai desktop assistant that uses Gemma 4 for screen analysis, voice memo transcription, and meeting transcription — all through llama-server. Everythi

daily reminder not to trust benchmarks and run it yourself. claimed e2e speedup is ~40%, forwards are ~140% faster I would wager that compared to a naive kernel anyone can write it's more in the range

https://github.com/mohsinkaleem/agent-mini.git A minimal 3k lines, local-first AI agent you can actually understand and extend. Optimized for smaller local models like qwen 3.6 4b or 9b pip install ag

Most agent memory setups run a model call on the way in. Something reads the turn, decides whether it's worth keeping, rewrites it into a 'memory', tags it with a type and an importance score. That's

There are a lot of LLM benchmarks but few, if any, harness benchmarks. I am thinking this would be a really good community project to build one. End goal: a leaderboard of harness performance (multipl

Hey. I'm very limited with brain capacity and I'm new to this. If you have experience, tell me which local model fits the best, what makes best blueprints etc. For example I want to spawn enemies usin

Come on guys, we could get deepseek cheaper with more usage directly with them if we wanted. WHERE IS KIMI K3 WITHOUT EXTRA USAGE NEEDED !!? This is pissing me off like crazy submitted by /u/Other_Che

Hey everyone, I’ve been building Speechfony - a desktop app for reading PDFs (and EPUBs) with offline text-to-speech. Open a document, listen sentence-by-sentence with highlighting, or export selected

Inspired by a post from u/giveen I motivated claude (no patinence on my side to work through everything myself) to help me get DS running on my MacBook M5 Pro 64GB and it exceeded my expectations.. be

I'm mosty interested in 128-192GB VRAM with 128-256GB RAM to spare, so SSD streaming is basically not even necessary. Seems only FP4 is supported, so older hardware will likely be slow - no Unsloth GG

https://github.com/yhfgyyf/vllm-deepseek-v4-sm89 I couldn't believe that someone actually got vLLM working with this particular set of GPUs, but here it is. The video is from right after I got it work