CompanyAnthropic8 recent entries9 Oct 2026Qwen 3.6 35B appreciation postReferring to specifically Unsloth's UD_Q4_K_XL quant because that's going to be a question, and is relevant regardless. It's old now. It's not great at coding medium sized or even small-ish projects. →10 Oct 2026Qwen3.8-27B on a single 3090: 140 tok/s on code with a custom megakernelI've been using Claude Opus 5.5 to speed up Qwen3.8-27B on my PC (rtx 3090), it wrote a CUDA megakernel that is 1.4-1.9x faster than llama.cpp depending on the task/context length. The results and cod
CompanyOpenAI8 recent entries5 Oct 2026PewDiePie getting banned twice by OpenAI while making a local model is top-tier comedy 💀So PewDiePie decides to fine-tune a local AI model called Ajax on his own computer. Pretty normal stuff for local model fans. To make his dataset, he uses OpenAI's API. OpenAI catches him using their →7 Oct 2026Running an open-source agent platform with no third party in the chain: what each piece needs (Ollama/vLLM, local embeddings, local OCR)Disclosure first: I work at Vstorm and I build AgenticOS, an open-source (Apache-2.0), self-hosted platform for building AI agents in a browser. This post was drafted with an LLM's help and edited by →10 Oct 2026Qwen3.8-27B on a single 3090: 140 tok/s on code with a custom megakernelI've been using Claude Opus 5.5 to speed up Qwen3.8-27B on my PC (rtx 3090), it wrote a CUDA megakernel that is 1.4-1.9x faster than llama.cpp depending on the task/context length. The results and cod→10 Oct 2026microsoft doubles down on local ai with nvidia, but with a costhttps://preview.redd.it/uyjw2lcrvkuh1.png?width=1536&format=png&auto=webp&s=ce991be22faec3de5cbaf39119c14782b6ed0af3 link to article TLDR; new microsoft surface and nvidia rtx spark laptops are starti→10 Oct 2026Engineer / developer observations of Gemma4-31B, Qwen3.8-27B, and 6.1-Sol for software engineering workFor what it's worth. I'm a mechanical engineer by education and software engineer in practice over the past ~20 years. Playing around with local models has been a recent hobby. Today I decided to do a→10 Oct 2026Basalt: Flash-Next at 665 tok/s structured, 354 prose on a 5090 + 5060 Ti (2.6x Strata)Basalt: Blackwell inference engine Hi all! So over the last week I've been working on a Strata fork that's heavily tuned and can achieve throughput up to 2.6x what Strata usually does on the same weig→11 Oct 2026'Strata' for GLM5.3 Flash is here for some! Project MayaI stumbled across this as I was currently having glm5.3 flash run only around 10tok/s basically unusable. With Maya i am running 30 tok/s now. GLM5.3 feels even with this low quant like a much more en→13 Oct 2026How far are we from a local model being able to optimize it's own runtime stack?A lot of projects have come up recently consisting of specific hardware config specialized forks of existing conferencing engines, but they all seem to be made using Claude or OpenAI's biggest frontie
CompanyGoogle8 recent entries26 Sept 2026For the longest time I’ve felt this sub should have a pinned section where a detailed post about each model should get featured.For instance whenever a model comes out, what’s the best engine to run it, the best harness and absolute minimum you need to get same or near same re results that the benchmark of that model claims. A→27 Sept 2026Qwen, where's the small stuff? (1B/2B/4B)I know Qwen is a key player in the local LLM space and has consistently introduced truly impactful technologies—like n-gram in Qwen-Next and the recent Qwen 3.8 27B, which is an amazing local model. H→2 Oct 2026Inference Engineering for DummiesHi all! I am a former SWE who has recently transitioned into inference engineering. I launched a side hustle a few months back and I've just taken it full time due to excessive demand. Its been such a→4 Oct 2026Poor People Vulkan GPUs listHelp with this list. Give me your recommendation on 'not supported anymore' GPUs. Looking for budget and Vulkan friendly options. Most of the GPU are not supported by latest CUDA / ROCm. Often with so→6 Oct 2026Introducing EmbeddingGemma 2: A best-in-class open model for natively multimodal embeddings | Googlehttps://huggingface.co/google/embeddinggemma-2 EmbeddingGemma 2 is an open multimodal embedding model built by Google DeepMind which maps text (incl. code), images, video, and audio inputs—and combina→6 Oct 2026google/embeddinggemma-2 · Hugging FaceEmbeddingGemma 2 is an open multimodal embedding model built by Google DeepMind which maps text (incl. code), images, video, and audio inputs—and combinations thereof—into a single, unified 768-dimens→8 Oct 2026GPU - Vulkan llama.cpp benchmarks sorted by price to performanceThis table to help anyone looking to build a budget Data Center homelab. I copied the bulk of value based, mid level, decent speed results GPUs and feed it to AI or SI and here are the recommended res→12 Oct 2026Explore DNA with DeepMind’s AlphaGenome and in-browser MedGemma 1.5 4B via WebGPUHey everyone, Sharing an educational tool called AskCyph Health Explorer to explore how single-letter DNA mutations lead to disease, how they alter 3D anatomy, inheritance probabilities, and available
CompanyMeta8 recent entries6 Oct 2026[Benchmark] Running Local LLMs on Orange Pi 5 Plus (RK3588, 16GB): Ollama Tok/s, NPU Offloading, Core Pinning & ThermalsDisclosure: This unit was provided free of charge by Orange Pi for testing. No editorial review, no preconditions, no script. All data, bottlenecks, and thermal behavior are reported directly from har→7 Oct 2026Why is ik_llama.cpp said to be faster than Mainline? On my hybrid multi-GPU rig, Mainline easily beats itI constantly see recommendations saying that ik_llama.cpp (ikawrakow's fork) is the undisputed king of hybrid CPU/GPU offloading and MoE performance. However, every time I benchmark it against mainlin→8 Oct 2026GPU - Vulkan llama.cpp benchmarks sorted by price to performanceThis table to help anyone looking to build a budget Data Center homelab. I copied the bulk of value based, mid level, decent speed results GPUs and feed it to AI or SI and here are the recommended res→10 Oct 2026Qwen3.8-27B on a single 3090: 140 tok/s on code with a custom megakernelI've been using Claude Opus 5.5 to speed up Qwen3.8-27B on my PC (rtx 3090), it wrote a CUDA megakernel that is 1.4-1.9x faster than llama.cpp depending on the task/context length. The results and cod→10 Oct 2026Qwen Flash Next @ 137 tok/s & 3,497 tok/s Prefill w/ 512k context on a 5090, 192gb ram, Windows Build, comparing Strata and InfernixStrata vs Infernix on the same model, RTX 5090 — A/B at 262k and 512k context Hi folks, another inference engine post for your feed. I was blown away by Strata, but saw someone else post about Inferni→10 Oct 2026Qwen 3.6 35B A3B: 131K context + vision on 6GB VRAMRunning HauhauCS/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive Q4_K_M with llama.cpp at ~600 tok/s prefill and 23 tok/s decode, 131k context window, Q8 KV cache - on an RTX 2060 6GB + 32GB DDR4 RAM. →10 Oct 2026Improve token per second without touching quantSpent the past month tweaking and experimenting with many different numbers to achieve 30tps. Hardware: -Rx6700xt 12gb vram (AMD) -2x16 ddr4 3200 ram -r5 5600x -llama.cpp vulkan sdk -window11 (no wsl →13 Oct 2026How far are we from a local model being able to optimize it's own runtime stack?A lot of projects have come up recently consisting of specific hardware config specialized forks of existing conferencing engines, but they all seem to be made using Claude or OpenAI's biggest frontie
CompanyMistral8 recent entries28 Jul 2026I got Kimi-k3 running.....Results: prompt eval: 40 tokens / 97.5s → 0.41 tok/s eval: 400 tokens / 1769.9s → 0.23 tok/s total: 440 tokens / 1867s (31 min) Prompt: 'Write a C++ function that reverses a linked list in place. Expl→19 Aug 2026GLM-5.3 is out on AA, and I'm fed up with their Intelligence/cost plotI think AA's intelligence/cost plot is seriously misleading, so I decided to make my own. Their plot is in the second image. All points are at max thinking. All intelligence index scores are from AA. →23 Aug 2026Your model fits in VRAM. Your context doesn't. The KV cache math nobody shows you.Spent way too long confused about why an 8B at Q4_K_M, about 4.9GB on disk, would OOM on a 12GB card. The weights fit twice over. The answer is the KV cache, and the math is simple enough that you can→11 Sept 2026Detecting hallucinations in local models without eating VRAM: What we learned testing 1.5B to 120B modelsHey everyone, If you run local models via Ollama in production or personal projects, you've probably run into the hallucination problem: how do you know when a model is hallucinating without burning e→12 Sept 2026For those of you forced to only use open models from Western labs in production, what are you deploying?First off, I know that GLM, Qwen, and DeepSeek absolutely dominate in terms of SOTA Open Source models, and that’s what I use in my personal projects and for school, however, I’m also responsible for →14 Sept 2026DeepSeek engineer relections on RSI - burying my talent to yesterdayNote - This is translated from the actual blog link right at the bottom. A few days ago, DeepSeek v4.1 was released. It raised the ability of small models to a new level. AI is improving much faster t→30 Sept 2026Follow-up: my native Rust + Vulkan Transformer training backend — 14 days later, now 14 parity-verified architectures and full PEFTFollow-up to my post from about two weeks ago. A lot has changed since then, so I wanted to post an update on where the backend is now. Where the green architectures stand When I posted last time, 7 a→12 Oct 2026QLoRA on ExLlamaV3 Quants - Train @ 2-8bpwEXL3-QLoRA I've built this over the last few months and I think it's stable enough to share now. A fork of ExLlamaV3, this is a native ExLlamaV3 training implementation with plain PyTorch, without tra
CompanyxAI5 recent entries15 Apr 2026need a grok/neno-banana like img2img generation colab cellA r/StableDiffusion thread where a user seeks a Google Colab notebook cell that replicates the img2img generation style or workflow of tools like 'grok' or 'neno-banana' — likely referring to specific→26 Jul 2026Will prices finally go down?I am seeing more and more videos as posts about how OpenAI is in complete financial ruin, Anthropic isn't much better. Their expenses go with the revenue they make etc etc. Meta made big investments i→23 Aug 2026DeepSeek V4 Flash on an M2 Ultra: repacked to 141 GiB losslessly, smaller than the Q4 GGUF, at 25.8 t/s (42 t/s peak)This is one more vibe slopped custom optimization for, in this case, my hardware (m2 ultra 60 cores, 192gb). It is just a fork from llama.cpp with a few changes, it achieves: - DeepSeek V4 Flash, no k→5 Sept 2026NInfer vs llama.cpp vs vLLM: quality + speed comparison for Qwen3.8-27B NVFP4 on RTX 5090I've been running Qwen3.8-27B as a local inference server for a production content intelligence pipeline (HVAC industry stuff, lots of long-context retrieval and structured extraction). I have been wa→9 Sept 2026SOTA ImageGen Locally NVIDIA Cosmos3(64B) INT4 quants CUDA/MLXCosmos3 INT4 T2I + I2V on Apple Silicon — code, weights and a Grok comparison GitHub - https://github.com/gtrg55/cosmos3-quant-mlx-cuda HF weights - https://huggingface.co/JuliaML/Cosmos3-Super-Text2I
CompanyDeepSeek8 recent entries3 Oct 2026I built Ninfer 4080 for 16GB class GPUsHi everyone, TL/DR I created NInfer 4080 to run ISTA-DASLab-Qwen-3.8-27B-GSQ at 100k context on an RTX 4080 16GB GPU using way more of the hardware capabilities (max overall: 2720 tok/s prefill, 262 t→5 Oct 2026Which model, which harness? I have data for you.I keep testing harness & model setups on real tasks: basic math, vision, computer use (reading emails, navigating stores) and coding. I created a benchmark and a leaderboard. You can see it here: http→5 Oct 2026Hardware Recommendations for around £6000 to £7000After some recommendations for hardware (I'm starting from nothing), in the region of £6-7k. I will be mainly looking to use it for coding and have found Deepseek V4.1 Flash or Qwen3.8 27b good and so→6 Oct 2026Local AI World Model Part 2 - Deep NN to turn Images into Playable Characters, with prompt switching mid rolloutLast time when I posted on this subReddit to share my work, the response almost made me cry because a tiny demo got so many people talking about this In the past 2 months Ive been training a new model→6 Oct 2026[Benchmark] Running Local LLMs on Orange Pi 5 Plus (RK3588, 16GB): Ollama Tok/s, NPU Offloading, Core Pinning & ThermalsDisclosure: This unit was provided free of charge by Orange Pi for testing. No editorial review, no preconditions, no script. All data, bottlenecks, and thermal behavior are reported directly from har→7 Oct 2026Local Qwen 3.8 27B vs DeepSeek Flash API: Is local good enough?Running a model on your own machine used to be a privacy story with a quality tax. On this generation that trade has narrowed to where we can state it plainly: for daily work, the local model is good →10 Oct 2026Qwen Flash Next @ 137 tok/s & 3,497 tok/s Prefill w/ 512k context on a 5090, 192gb ram, Windows Build, comparing Strata and InfernixStrata vs Infernix on the same model, RTX 5090 — A/B at 262k and 512k context Hi folks, another inference engine post for your feed. I was blown away by Strata, but saw someone else post about Inferni→10 Oct 2026Qwen 3.6 35B A3B: 131K context + vision on 6GB VRAMRunning HauhauCS/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive Q4_K_M with llama.cpp at ~600 tok/s prefill and 23 tok/s decode, 131k context window, Q8 KV cache - on an RTX 2060 6GB + 32GB DDR4 RAM.
CompanyNVIDIA8 recent entries8 Oct 2026GPU - Vulkan llama.cpp benchmarks sorted by price to performanceThis table to help anyone looking to build a budget Data Center homelab. I copied the bulk of value based, mid level, decent speed results GPUs and feed it to AI or SI and here are the recommended res→10 Oct 2026No more RTX 5090Nvidia reportedly halts GeForce RTX 5090 production in favor of AI data center and professional GPUs — impending supply drought expected to drive up prices, RTX 5080 24GB rumored as new gaming flagshi→10 Oct 2026microsoft doubles down on local ai with nvidia, but with a costhttps://preview.redd.it/uyjw2lcrvkuh1.png?width=1536&format=png&auto=webp&s=ce991be22faec3de5cbaf39119c14782b6ed0af3 link to article TLDR; new microsoft surface and nvidia rtx spark laptops are starti→10 Oct 2026Basalt: Flash-Next at 665 tok/s structured, 354 prose on a 5090 + 5060 Ti (2.6x Strata)Basalt: Blackwell inference engine Hi all! So over the last week I've been working on a Strata fork that's heavily tuned and can achieve throughput up to 2.6x what Strata usually does on the same weig→11 Oct 2026vllm-ascend updates (GLM5.3-flash) on dual 310p Ascend cardsAlright, I learend a few valuable lessons since I posted about a week ago, and I've made more strides with the DDR4x Ascend 96GB 310P cards I purchased, so here we go, meet the new friendly and less v→11 Oct 2026'Strata' for GLM5.3 Flash is here for some! Project MayaI stumbled across this as I was currently having glm5.3 flash run only around 10tok/s basically unusable. With Maya i am running 30 tok/s now. GLM5.3 feels even with this low quant like a much more en→11 Oct 2026I trained a 102M recursive BitNet-v2 model from scratch: 64K context, trained on less than 5B tokensDISCLAIMER: the post and the model card was made with the assist of AI. Hiya, I’m releasing Recursive BitNet N-Gram 102M, a small experiment combining ternary weights, shared transformer layers, and h→12 Oct 2026A Record of the Process of Increasing MFU from 21% to 50% While Pre-training 21B MoE Using a Single GH200 Chip (Includes Code and Logs)I am an independent researcher. Due to budget constraints, I shifted my 21B MoE pre-training—which I had originally planned to run on 64 H100s—to a single NVIDIA GH200 ($2.29 per hour), and I have com