Knowledge catalogue
Hello everyone Yesterday I gave a report on Gemini bugs and the techniques I learned on Gemini so far with the Engineering Prompt and interestingly today I got a very interesting and controversial ans
For what it's worth. I'm a mechanical engineer by education and software engineer in practice over the past ~20 years. Playing around with local models has been a recent hobby. Today I decided to do a
Hey everyone! 👋 Feeding multi-gigabyte files (SQL dumps, videos, audio, PDFs, and spreadsheets) directly into LLMs often causes context window overflow, GPU VRAM bottlenecks, massive token bills, and
Hello all, I am wondering if a transition from a Ph.D. in electrical engineering (Quantum optics/photonics) to a job in ML is a reasonable aspiration. Personally, I have extensive software development
Been benchmarking DSv4 Flash 0731 on a Flow Z13 (Ryzen AI MAX+ 395, Radeon 8060S / gfx1151, 128GB LPDDR5X) for the past week. Figured I'd share what actually works and what doesn't — there are a lot o
Hi all! I am a former SWE who has recently transitioned into inference engineering. I launched a side hustle a few months back and I've just taken it full time due to excessive demand. Its been such a
Hey guys, Last time I tested Qwen3.8-Flash-Next on its own. This time I put three Qwen3.8 checkpoints through the same 10 tests on the same RTX PRO 6000: RadixArk/Qwen3.8-27B-NVFP4 (dense) orcarouter/
This r/MachineLearning discussion thread centers on a hands-on workshop covering context engineering — the process of designing and optimizing the information an AI agent sends to and receives from a
🚀 We just built our first real-time implementation of Graph Engineering, inspired by our experience building graph tooling used by 4,000+ developers. 🔗 Repo: https://github.com/CodeGraphContext/grapha
This r/MachineLearning discussion post examines LLM jailbreaks through the lens of social engineering, arguing that the psychological vulnerabilities found in LLMs are not random artifacts but structu
A community researcher shared on r/MachineLearning how they extended Chatterbox TTS — Resemble AI's open-source, 500M-parameter model — to support 8 Indian languages using LoRA (Low-Rank Adaptation),
*AI Systems Performance Engineering* by Chris Fregly is a ~1,000-page book (published December 2025) covering GPU/CUDA kernel tuning, PyTorch optimization, distributed training, and high-throughput...
AI Engineering from Scratch is an MIT-licensed curriculum: 523 lessons across 20 phases, from linear algebra and backprop to transformers, LLMs, agents, and production serving. The code is stdlib-firs
Someone prompted different LLMs to generate CAD code for a bridge under fixed constraints (2-foot span, under 500g filament, 18-hour print limit), printed them, and load-tested them to failure. The re
Does anyone know of a decent reverse engineering harness/workspace setup for local models that I can just point the model at and have it go to work until it's reversed most if not all of the functions
First time trying something like this. I asked chatgpt and it told me there are mcp server plugins for both ghidra and ida pro as well as RE-MCP which has support for both. I could run that on my low
I'm a teacher looking into running an LLM locally. I've been using ChatGPT for a lot of this work, so I started looking at Ollama because I understand it's designed to run LLMs locally. That said, I'm
I’ve been building Catalyst around a capability I want engineering agents to have: An AI agent can read a program and make a good guess about what matters. Catalyst lets it measure what actually matte
ninfer, dwarfstar, Splash, llamAmpere, gufo, etc. We've all seen them popping up, great tok/s, people loving them. Forks of llama.cpp or another engine, or made from scratch. For better or worse, the
I've been running Qwen3-0.6B on the M5Stack LLM-8850 card (Axera AX8850 NPU, 24 TOPS, 8GB LPDDR4x) hosted by a Raspberry Pi 5 — as a llama.cpp backend. The problem: the vendor stack requires convertin
Hi guys, I am pretty happy to announce MegaCapybara. Purpose build engine for RTX5090 that is focused on Qwen3.8 27B (more will come later). GITHUB (Engine) HUGGINGFACE (weights) Why ? 1. It beats Nin
We built an engine, Kyojin, on top of ExLlamaV3 for Strix Halo (gfx1151, ROCm), and packed two 300B-class MoE models so each fits one 128 GB machine. First release, all measured on Ryzen AI Max+ 395.
There seems to be a whole category of extremely narrow inference runtimes appearing: Strata, ninfer, DwarfStar, Splash, llamAmpere, gufo, etc. They deliberately give up the thing llama.cpp/vLLM are gr
It’s probably a bit niche and the models are a bit old at this point, but after reading about Ninfer a few months ago I did my own small vibe coded engine project for my 5080 Laptop GPU. Initially I t
WARNING: I think I made a mistake with omlx, and its results below seem to be without any kind of acceleration. I am rerunning them with Lightning MTP and ANE support for prefill. Will update the post
Hey r/Ollama, If you run local models on mid-range hardware, you probably know how fast VRAM disappears when you try to set up document memory or local RAG pipelines. I built Hillock (https://github.c
came across a pretty interesting technical breakdown of chatgpt's newly launched 'intelligent ui' feature, and thought this subreddit might find it interesting. for anyone unfamiliar with the concept,
There are way too many forks to keep track of, so I've gotten lost. As far as I can tell, Radiance VLLM is best for models that fit in GPUs while some form of llama.cpp is probably best for MOE RAM-sp
# the What An engine to run Gemma 4 31B on blackwell under massive concurrency and rather specific workload patterns. I've been waiting for someone to do ninfer but for gemma, and, well, ended up havi
https://preview.redd.it/nulsv53o8vqh1.png?width=4500&format=png&auto=webp&s=74765dbd409f4c221640f9f6000a685f6fdbb242 Spent weekend benchmarking the Splash engine (by Incoai) and extending its architec
Hi everyone, Sharing a personal project exploring the minimal bare-metal footprint required to run an autoregressive LLM. Instead of relying on large runtimes or compiler abstractions, I wrote an infe
Working on a resource dashboard, rich logs, lightweight 64mb cap, all local, scales on network API endpoints via collector, supports multiple engines (llama, strata, custom cuda engines, unsloth, LMS)
I forked llama.cpp's server into AgrillaMoE, a dedicated build for Qwen3.6-35B-A3B (~A4B) with Unsloth quants. On a (vant.ai) rented V100 16GB with the 2-bit UD-Q2_K_XL quant it generates at ~57-60 to
Wanted to share a local memory engine I built called Hillock. The main problem I had with standard local RAG was that running document parsing and vector search chewed up so much VRAM that my actual O
I think most people are sleeping on this inference engine. I tried multiple llama.cpp forks and none of them comes close to the inference speed of Strata. Initial version had some bugs with kv cache,
Hey guys, After my CPU-only to 96GB VRAM test, I tested Qwen3.8-Flash-Next across llama.cpp, SGLang and FreeToken on the same workstation. This time I wanted to see what changes when you keep the hard
I'm one of the developers. We said in August it would go open source in September and it did last night. MIT or Apache-2.0, pick one. The repo you see is our internal repo, kernels included, so from n
Hi, everyone, I’ve just released VoxGen, a lightweight native inference engine for VoxCPM2, written in Rust and using Vulkan compute instead of Python/PyTorch/CUDA. Why VoxGen? The main reason I start
For years, the JVM has watched the AI revolution from the bench. Every model, AI framework, every breakthrough, built with/for Python. jinfer is an inference engine built for the JVM from first princi
Note - This is translated from the actual blog link right at the bottom. A few days ago, DeepSeek v4.1 was released. It raised the ability of small models to a new level. AI is improving much faster t
Hi everyone, it's been a while since I posted so here's an update on what the Lemonade community has been up to this summer. Our overall mission is to enable local AI builders with everything they nee
Disclosure: I’m the author and maintainer of QuarkStar. I built QuarkStar, a small native inference engine inspired by Antirez’s DwarfStar. QuarkStar currently supports: Qwen3.6-35B-A3B, using the sam
For 6 months now I've been trying to make agentic coding work for me, using Pi and a handful 30-120B models (Qwens, Nemotrons, Leguna...etc). I'm not greedy either, I stick to decent quants, never qua
Its a custom Swift/Metal inference engine that runs Gemma 4 26B-A4B-IT on M-series Macs with very low RAM. It uses ~2GB instead of ~14 GB. The result is reportedly 5–6 tok/s on an 8 GB M2 MacBook Air
Kimi K3 by Moonshot reached the frontier as an open-weight model. Artificial Analysis ranks it fourth of 580 models, behind only Claude Opus 5, Fable 5, and GPT-5.6 Sol. Moonshot released more than th
This is an open source and free tax engine which scored 96% on TaxCalcBench [highest ever recorded score till date] surpassing fable 5 and sol with just sonnet 5. The only 2 cases where it missed, it
Hey r/LocalLLaMA, Like many anime fans, I've always been frustrated by traditional MT engines (like Google Translate or base DeepL) when dealing with raw Japanese anime: - They completely butcher Japa
Hey all. We've spent the last weeks getting Qwen3.8-Flash-Next (125B MoE, 6B active) to run properly on one AMD Strix Halo box (Ryzen AI Max+ 395, 128 GB). Tonight we're releasing both the 95 GB EXL3
Over the past few months I have been building a CPU-first inference engine from scratch in pure C99 (no Python, no CUDA, no BLAS, just GCC and make). The focus has been running 1.58-bit ternary models
I maintain xberg, an open-source (MIT) document extraction engine, and v1 is out. Sharing here because a common piece of a local Ollama RAG setup is 'get clean text out of my PDFs/Office files/images,
This post discusses implementing Physics-Informed Neural Networks (PINNs) to solve the 3D heat equation within a C++ WebAssembly engine for real-time aerospace applications. The work combines machine
A Reddit user on r/StableDiffusion built a locally-running RAG (Retrieval-Augmented Generation) engine designed to eliminate the need for manually crafting lengthy, model-specific image generation pro
It seems to use a 96-bit LPDDR5X memory bus, instead of the previous 64-bit wide busses. Considering it's on 2nm, that's expensive silicon. That should result in around 115 GB/s memory bandwidth. A20
mentria.ai is a browser inference engine I've been building solo, from scratch in WebGPU/WGSL. This week it crossed a milestone I had been chasing for a while: a 27B one-bit model answering at up to 3
This post was not written by a clanker. Hey guys, I'm a comp sci major who wanted to introduce a cool project I built for quantizing models to ternary (1.58 bit) with as minimal of loss as possible, a
I only started using some vLLM forks recently in a 4 x 48GB 4090 system. DS4F - ~5000pp/180tg (DSpark) Qwen3.8 Flash next - ~7500pp/135tg (MTP) This is amazing, like having the API in my house. But it
Been building this for a few months, mostly for myself, and it just got a proper release so figured I'd post it. It's a native GGUF inference runtime with OpenAI/Anthropic-compatible APIs and a chat U
In medium reasoning mode, it both scores higher than the 3.6 version, AND is very much more efficient (almost half requests needed, and a third less tokens generated) - at DeepSeek v4 Flash 3107 MXFP4
Qwen3.8-27B is finally here, and NInfer already has Day-0 support! Weights: https://huggingface.co/neroued/Qwen3.8-27B-NInfer Just update to the latest source and give it a try. On a single RTX 5090,
This post describes a project where a developer created a searchable database of 49,000 sample images featuring Anima AI characters and artists. The tool appears to be designed to help users find and