Knowledge catalogue
Around 3 weeks ago, I posted on this subreddit about how Jev used my one-year-old architecture, og post: https://www.reddit.com/r/LocalLLaMA/s/VpuMJt577S Then, just after that, I introduced Laya, the
For what it's worth. I'm a mechanical engineer by education and software engineer in practice over the past ~20 years. Playing around with local models has been a recent hobby. Today I decided to do a
A conversational AI plush toy where nothing leaves my network. Everything runs on my LAN: STT: Whisper LLM: Hermes 3 Llama 3.1 8B (Q8_0) TTS: Kokoro Memory: per-person (LangGraph). It knows who it's t
I wanted to compare gemma 4 31b qat q4 and qwen3.8 27b iq4_nl, so I run SWEBench verfied on them. The original gemma 4 chat template doesn't work with preserve reasoning, so google released an update
I was looking into finalizing my daily driver and it was not clear which one to chose. I have a 5060ti 16GB so I picked the following contenders, which seem to be working at a reasonable speed on my h
Hi All: I have an Asus ESC 4000 G3 with 128 GB DDR4 RAM - I tried putting in my V620s but couldn’t put more than 2. Sadly, pivoted to T4s, these are 72 watt passive cards and I thought I could use the
I ran a small test for Dory to see whether a local model could update one fact in a synthetic notes file without losing any of the others. I asked Gemma and Qwen to rewrite the file, return a patch, o
Spent the past month tweaking and experimenting with many different numbers to achieve 30tps. Hardware: -Rx6700xt 12gb vram (AMD) -2x16 ddr4 3200 ram -r5 5600x -llama.cpp vulkan sdk -window11 (no wsl
So, while doing a search to read the model card again, and because I didn't memorize the huggingface URL, I saw an AI 'answer' at the top of the search, stating that while it natively supports a ~244k
https://preview.redd.it/uyjw2lcrvkuh1.png?width=1536&format=png&auto=webp&s=ce991be22faec3de5cbaf39119c14782b6ed0af3 link to article TLDR; new microsoft surface and nvidia rtx spark laptops are starti
Hey everyone! 👋 Feeding multi-gigabyte files (SQL dumps, videos, audio, PDFs, and spreadsheets) directly into LLMs often causes context window overflow, GPU VRAM bottlenecks, massive token bills, and
Nvidia reportedly halts GeForce RTX 5090 production in favor of AI data center and professional GPUs — impending supply drought expected to drive up prices, RTX 5080 24GB rumored as new gaming flagshi
DigUp is a free Mac app that runs Google DeepMind’s new EmbeddingGemma 2 locally over your own files. The model puts text, images, audio and video in one space, so you describe what you remember and l
Running HauhauCS/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive Q4_K_M with llama.cpp at ~600 tok/s prefill and 23 tok/s decode, 131k context window, Q8 KV cache - on an RTX 2060 6GB + 32GB DDR4 RAM.
Strata vs Infernix on the same model, RTX 5090 — A/B at 262k and 512k context Hi folks, another inference engine post for your feed. I was blown away by Strata, but saw someone else post about Inferni
I've been using Claude Opus 5.5 to speed up Qwen3.8-27B on my PC (rtx 3090), it wrote a CUDA megakernel that is 1.4-1.9x faster than llama.cpp depending on the task/context length. The results and cod
I am developing a Windows-based local AI virtual spouse application that supports adult roleplay. My goal is to run it without a server or external AI APIs. I am looking for existing open-source modul
Been seeing quite a few posts about Strata lately, so I figured I'd give it a shot on my RTX PRO 6000 and I am very impressed. Most of the benchmarks I've seen are using IQ2 or IQ3 quants, so I wanted
https://typesafe.ai/blog/series-ai Meanwhile one of the reddit members apparently developed the same architecture a year ago: https://www.reddit.com/r/LocalLLaMA/s/8zQXqCNDlb Looks like ai valuation n
Hey everyone! 👋 After months of building, we’re finally opening the first spots for **Many Agents**, a new product from Dome. The idea is simple: **What if you could create your own team of AI agents
https://preview.redd.it/7sknv6omnguh1.png?width=1023&format=png&auto=webp&s=4aef1dc464d008b00f104519eb5399ba6132ad69 Want to try? Just go here and click 'New seed' a lot. You will get different storie
I made BigMoeOnEdge+ (BigMoeOnEdge-plus), a fork of Helldez's BigMoeOnEdge, the engine that streams MoE experts from flash so models bigger than RAM can run on-device. It adds a full Android chat app
For those of you who have been wanting to utilize your dual Spark cluster for image/video gen with ComfyUI - today is your day. I started this personal project back in April of this year, and it’s at
I checked the site, and it didn't say anything about mlx only on the ollama models listing. So why is this restriction in place? ollama pull embeddinggemma-2 pulling manifest Error: this model require
I posted on here a while back asking for advice on upgrading to an Epyc system. Thanks to all who commented. I’ve taken the plunge and ordered: - Epyc 7443 CPU (24 cores, 48 threads, 128 lanes) - H12S
Blog Post : ML Drift: Next-Gen GPU AI/ML Inference at the Edge - Google Developers Blog The Google AI Edge Team is excited to announce the open-source release of ML Drift, our high-performance, cross-
It seems Dario's 'too powerful for you users' strategy is paying off: we have two open models at the top of the leaderboard, surpassing every single model from Anthropic. Open source prevails. Even Mi
Disclosure: I work at H2O.ai. We released H2O-Lightning-4B, an open-weight (Apache-2.0) model for the 'decisions API' style of inference that Jev made popular: you send a state plus typed questions (p
Hi everyone! So far in 2026 I have been iterating and evolving a research system I designed to help me sift through things that LLMs all agree on that may not be correct, and to give me actual confide
ALHR - Adaptive Learnable Hierarchical Routing, uses static binary trees and learnable functions to minimize the amount of keys to be read. It does use a dense teacher while phase 1 of training howeve
I built MaRN (Mapping Networks), a PyTorch library that lets you optimize a compact latent representation instead of directly training every model parameter. Some results from my current benchmarks: M
Disclosure: I'm the developer. The app is called Citenook. I wanted to ask questions across a few hundred PDFs and notes with Ollama, but every setup I tried either needed a lot of glue code or gave a
Last month I created a portable memory format as a .txt file as a small project. I named it as: Memory Journal, in a shape of a skill card. It's essentially a large prompt wrapped in a text file. Init
Live results, coding benchmarks included, agentic benchmarks included, domains and other classifications filters included. (Spoiler: DeepSeek-V4-Vision-Exp rules, but other models have their rule area
In the dawn of the LLM craze, when wild Llama2's roamed free and 4k tokens was considered a fairly long context for local models, I forked an interactive fiction and MUD library called Tale to experim
It's been a long time coming, and I should have done this in the first place. But LumaBrowser is now open source. Since I first released it a year ago, it's had several thousand installs and an absolu
Speculative decoding accelerates autoregressive generation in large language models. In each drafting stage, a lightweight draft model proposes tokens that the target model subsequently verifies. With
Conditional memory architectures such as DeepSeek Engram use input n-grams to look up learned embeddings, expanding the capacity of large language models (LLMs) with limited additional computation. Be
Training large language models (LLMs) is resource-intensive, and adapting them for diverse deployment scenarios with varying computational constraints remains challenging. While elastic architectures
Referring to specifically Unsloth's UD_Q4_K_XL quant because that's going to be a question, and is relevant regardless. It's old now. It's not great at coding medium sized or even small-ish projects.
(Don't judge by the screenshot, the cache is cold. It hits 24+ tok/s with a warm cache!) About two months ago, I made a post here asking whether predicting which MoE experts would be used on the next
Qwen-Image-2.1-Turbo, create and edit images in just 8 denoising steps! Open weights now available! Built on Qwen-Image-2.1, Turbo is an accelerated checkpoint on the same 7B visual generation archite
LemonSeed Studio is an iPad editor/IDE with on-device inference on an AMD GPU in a Thunderbolt enclosure. It embeds the unmodified upstream Linux amdgpu + amdkfd driver (mac_linuxgpu) as a PCIDriverKi
I wanted the uncensored Qwen3.8-27B (llmfan46's Heretic build, MTP head preserved) on my RTX 4080 with MTP on (because I've been testing TONS of configs, and realized that would be the one) And none o
Hello all! Thank you to everyone that contributed their feedback and notes for LlamAmpere the last time I posted. I've continued to chip away at improvements for the 3090 crowd -- this release is mode
ik ik 'another harness D:'. so... i spent the last 6 months of my life building and ironing out every single detail and bug through almost every route with extensive testing across 3 machines, daily,
I have rtx 4090 and 64 gb of ddr4 ram. This is enough to fit smaller quants of qfn, but I found them not great and switched back to qwen3.8. However, there are option that bother me a lot, buy huawei
HauhauCS ships their uncensored Qwen3.8-27B as GGUF only. NInfer, a C++/CUDA wanted its own format. Now the same model that ran at 91.6 tok/s / 131K under llama.cpp does: 262K context (the model's ful
SlopSoup TV is a live, never-ending pixel-art TV network. Every script, character, voice, camera cut and schedule decision is made by models. It's made to be very absurd, dark and strange. Hardware: o
All anyone ever talks in this sub is about x new model having x benchmark scores and how well it can code. Or some new tech to increase t/s for qwen. So I wanted to go back to the roots of this sub an
A bit of backstory : i got lucky in August and was able to grab a CMP 170 HX 8GB for around 500 on alibaba. I tried later to get a second one for around 1k but my order got cancelled and prices went t
Youtu-Parsing-Omni is a compact (5B) omni-modal parsing model. Given a single input — a document page, a natural image, a chart / flowchart, a geometry figure, an audio clip or an audio-visual video —
I tested Mellum2.1-12B-A2.5B (Q8) locally using Pi and llama-server. All five tests were one-shot. Results were pretty mixed: - Pelican SVG, Browser OS, Minecraft: Poor results. - Bouncing Hexagon: Ph
Figure 1b from our paper Disclosure: I'm one of the authors (Microsoft). The paper, code, dataset are public and ThinkingBox is on Hugging Face OpenEnv as well. Raw evaluation trajectories are not rel
I've been building ANCHOR, a web automation agent that runs entirely on Ollama. You describe a task in plain language ('Register Maria Silva, IT department, contractor, and save'), point it at a site,
Preamble and disclaimer, Sample size: 8, at any size of form this research is just screwing around being writing it down. This was just a little,(well... big), curiosity test I wanted to run to see wh
Post title is slightly misleading, I don't think you can get these for $350 each anymore but they're still pretty cheap all things considered. They're Radeon Pro V620's which are older RDNA2 enterpris
UCLA’s Trustworthy AI Lab is running a tournament on October 16 where AI agents compete in Pokémon Showdown, Werewolf, Red Alert, and Honor of Kings. It’s open to everyone, including remote participan
Hi all, a bunch of performance improvements have been landed in audio.cpp. The biggest highlight is Higgs Audio TTS, which now runs with around 6 GB VRAM, a 48% reduction in peak memory usage compared
There are way too many forks to keep track of, so I've gotten lost. As far as I can tell, Radiance VLLM is best for models that fit in GPUs while some form of llama.cpp is probably best for MOE RAM-sp