Knowledge catalogue
Been building this for a few months, mostly for myself, and it just got a proper release so figured I'd post it. It's a native GGUF inference runtime with OpenAI/Anthropic-compatible APIs and a chat U
do you want some omni? here is omni for you 1. 🧭 Overview This repository hosts two checkpoints of the Realtime-Venus system: Realtime-Venus-Omni (Realtime-Venus-Omni/): the 9B audio-visual interactio
Baseten launched a new safety infrastructure standard alongside its Base Labs research arm on Wednesday, partnering with Hugging Face and Goodfire AI to build safety evaluation and monitoring infrastr
Thanks for the exceptional support (https://www.reddit.com/r/LocalLLaMA/comments/1wijo3e/i_literally_built_the_jev_architecture_one_year/) and for the dozens of requests to make a generic model, run b
MiniMax has open-sourced the terminal version of MiniMax Code: https://github.com/MiniMax-AI/minimax-code How can developers verify the content that encoding proxies read, send, and store? This is a t
MiniMax has released the source code for MiniMax Code’s terminal agent: https://github.com/MiniMax-AI/minimax-code https://preview.redd.it/st9izmm8zaqh1.png?width=3016&format=png&auto=webp&s=5853bc316
I had Astra run a multi-hour investigation into speeding up local MoE inference while keeping the model weights and quantization unchanged. The experiments covered Qwen3.6-35B-A3B and Qwen3.8 Flash-Ne
Saw this little dude in a Forbes article (https://www.forbes.com/sites/johnkoetsier/2026/08/18/american-humanoid-robot-launches-for-just-1688-delivery-this-fall-made-in-san-francisco/) This is definit
Hey r/LocalLLaMA, Prism-LM recently released its Bonsai 2 QAT models based on Qwen3.8, and they quickly gained traction. In our evaluation, the models strike a strong balance between throughput and qu
Hey community, Jovan from UkisAI here, We're the team behind Swift Qwen3.8 27B, the Qwen model with token usage and overthinking error improvements Our estimate is that we can make a great improvement
I've been running Qwen3.8-Flash-Next as my main local coding model from past few weeks. The file is 95.5 GiB and my Mac has 64 GB. It works because the routed experts stay on SSD and only get read whe
Looking at jev launch website and demo video on x.com…. it seems like it’s a very intelligent classifier with custom prompt and custom criteria instruction reading capabilities. It can do well defined
TypeSafe announced Jev on Tuesday: you give it data plus typed questions (yes/no, pick-one, 0–N scale) and it returns a probability for every option, crazy fast. I signed up and then refreshed my inbo
Ternary Bonsai 2 27B Tried it out and was excited. Gave it a prompt 'create a 3d little planet globe where I (player can walk around) and it has all these biomes to explore, the globe doesn't have to
We just released the first version of our Creative Writing benchmark, comparing 24 LLMs against human writers across 475 writing prompts. Creative writing is subjective, so the rankings aren't meant t
I let AI run my phone 60 real tasks, back to back, on the OnePlus I use every day - Best text model still failed 43% of them - Peak chip temp 98.2 C - 69% of the battery gone The benchmark is AndroidL
Basically the title. Training models take time and if a model was already in the process of pre-training/RL then it can't be re-cooked with any new architectural improvement that has been discovered i
Hey all, Henry from Cactus Compute here, I kinda wanted to share our latest model and get feedback from the family :) Needle 3 is a small foundation model for automation: you give it the functions you
Yoyo I renamed https://github.com/syv-ai/qwen38-27b-rtx3090 to https://github.com/syv-ai/HyperQwen/ to focus more on Qwen models and 'local' hardware in general, than just a single card and a single m
just saw some benchmarks on the omlx website for the m5 ultra (don’t know how official they are but they seem reasonable): Link For Qwen 3.8 27B q4 it gets 50 tok/s th and 1800 tok/s pp at8k context a
What I want to know is how your daily interaction routine looks like. Do you wake up, take your coffee/tea/drinkname.txt, have a nice morning, then click a button and somethings starts and creates som
We compared 20 model versions on 360 frozen joke prompts, with four jokes requested per prompt and model names hidden from our humor-trained judge. Fable 5 had the highest estimated score: 66.8 points
Everyone now talks about the architecture that's not auto regressive and does lightning fast probability prediction with a json schema. I worked on this literally one year back in March 2025, publishe
IFM released K2-Horizon-7B , diffusion augmented LLM at upto 5200 tps, with claimed lossless speedup. causal LLM architecture and adds a plug-and-play diffusion adapter alongside the autoregressive we
CPU : epyc 7262 MB : ROMED8-2T GPU : V100 16GB PCIE x6 I have built a server with six V100 GPUs. I am now testing it and plan to eventually run Qwen3.8-Next-Flash configured with TP2 and PP3. Because
UPDATE https://github.com/ggml-org/llama.cpp/pull/29009 MERGED More Gen AI coverage and frameworks integrations to minimize code changes New models supported: On CPU: Gemma-3n On CPU, GPU: Kokoro-82M,
I was testing this quantization IQ3_XXS from GSQ-RCO with PI. It is a heavy quantization case, the model is in IQ3_XXS and KV cache in (Q4_0, Q4_0). I asked it to make the flight simulator, using that
Hey Reddit. This is a post asking for advice / user experience. The goal is simple: deploy a small private server for a developer to run a harness that rivals/beats Claude Opus (in perf/intelligence,
The model is derived from Qwen3.8-27B, a 27B hybrid-attention causal language model (architecture unchanged), but uses ternary weights to shrink model size down to <6GB in size. According to the model
Hey everyone, Jovan from UkisAI here, a small lab building the tech to make tiny frontier LLMs possible (and doing it open-source!) The purpose of this post is simply to thank the community for all th
Anyone else trying out the new Union stealth model on openrouter? It’s pretty good for a free model, and of course I had to run tokenizer tests and training cutoff tests which established it’s the 99%
I posted something about a 9b model a few days ago. The real problem was 2 fold. Someone suggested the size was too big to prove it all out. It's a fair argument. I needed the model depth though. The
We've been tuning vLLM/SGLang/llama.cpp setups for a long time and got tired of the profiling part: nsys trace, open the GUI, squint, change a flag, repeat. The profilers assume a human is looking at
I find another new model at HF: https://huggingface.co/XingChen-AGI/Xing4.0-29B-A4B 'Xing4.0-29B-A4B is a next-generation large language model in the Xing series (formerly TeleChat), developed by Chin
hey folks, it's Merve from Hugging Face 🤗 with the latest cool things happening in between LLMs and 3D, we have written an illustrative explainer guide on 3D asset landscape, what 3D representation yo
Apple is considering offering an AI server built around 'M8' series chips and has discussed incorporating Nvidia networking hardware, The Information reports. The system would be sold to outside custo
Hey experts! I genuinely have this question and would love a general consensus from people actually using these models. LLMs have advanced a lot in benchmarks and in practical usage. Even GPT-4o and G
can you draw a picture of a flower in gimp? So I ran through this setup to install the MCP server suite from GITHUB, it's pretty easy to follow the install - and installs the libraries necessary to co
I've been working on something that started as the training backend for my Hierarchos architecture, but it has grown into a much broader project: https://github.com/necat101/Hierarchos-Native Hierarch
We gave two small LLMs a real customer-service task. One got 100/100. The other stopped with the job half-done. I wanted to test something more meaningful than benchmark scores: can a small model actu
A follow-up to my earlier post about small models on phones. Last time it was fixed tasks in a single browser. This is something else. Ministral 3 3B Q4_K_M in llama.cpp on a Samsung Galaxy S21, four
It's late at night in the lab, I've been working on a basic script for a virology project, and holy hell the safeguards have been pissing me off. Mirroring detectEVE data over rsync to my laptop by ma
It's an old Mitch Herdberg routine about tricky survey questions. But every time Qwen 3.8 spits out one of those sentences that is structurally correct but makes no actual sense, it pops into my head.
I let Qwen 3.8 27B 4bit quantized with 100K context window run autonomously for 63 hours (50 million+ tokens) to try to solve the RH. Of course it did not solve it, but the experiment still shows it's
Saw the new Introducing System One Models & Jev - TypeSafe AI Blog Jev model which just outputs probabilities given choices and I thought it sounded a something the Qwen reranker models could do? Then
Setup for one long agentic coding session (tools + vision) on a single 5090: Qwen3.8-27B RVN Heretic (ARA abliterated) at Q6_K, 159744 context, 1 slot, q8_0 K/V, DFlash2 speculative decoding, vision p
Achieved steady 15 tokens/second output and 100-120 promp processing per second with 12GB GPU (RTX 5070 SFF) with Qwen3.8-Flash-Next-GSQ-RCO-GGUF at 3 bpw (IQ3_XXS which maches BF16 on AIME25). Around
At last my small pebble of contribution has been merged. Thanks to CISC. Support for the model DFM Mimir 1B gguf here: https://huggingface.co/noctrex/DFM-Mimir additional optimization will be added in
https://preview.redd.it/e5wn8eyh7vph1.png?width=1080&format=png&auto=webp&s=5faeec866acc9e8eca8dc9e84a6661bee7bcf954 https://preview.redd.it/8ov5gl8j7vph1.png?width=1080&format=png&auto=webp&s=440320c
I am planning to buy hardware to run local models. I'm not sure what kind of hardware is best and what will I be able to run. I'm torn between getting a used M1 Max 64gb Macbook for €1500, M3 Max 128g
Follow up post of https://www.reddit.com/r/LocalLLaMA/s/nGMyKswrch. Thanks everyone who replied. I didn't change the specs. Might be loosing some of the memory bandwidth but will scale in future if I
What taste in humor do LLMs have? Do LLMs like certain types of jokes? Do they have strong opinions? Our report: LLMs have some overlap in humor taste, about 3/4 agreement but it ranges from 54.9% to
With the recent Nvidia Huggingface acquisition and frontier AI labs screaming about safety and putting guardrails everywhere, I think it's important that we have local models that aren't affected by a
I've been working on something for this community (and other self-hosted communities) for almost 5 months now. Personally, I would grade it as probably the closest open source equivalent of Manus and
https://reddit.com/link/1wi2zuc/video/rs1a4sx92xph1/player Hey everyone, I’ve been experimenting with local LLMs on Intel hardware, and I wanted a hyper-optimized, standalone CLI tool that didn't rely
I'm pretty sure it can be done with any model based on qwen4exp, which Qwen's next local models will be based on. You can use a quant that barely fits in VRAM and still run at the model's maximum cont
I implemented expert lookahead, a trick that gets a 10%+ performance improvement on MoE models running on low-memory macs (Qwen 3.8 flash in this case) while using expert-offloading/ssd-streaming (usi
So I would like to run qwen 3.8 27b locally for my ai agents, maybe even in parallel with other small LLMs (such as qwen3.5 9b, oss 20b etc..). What is the best hardware to do this? Not a video card b
Hey r/LocalLLaMA, We’ve released our full ShapeLearn GGUFs for Qwen 3.8 27B. Blog / Download models TL;DR 3.84 bpw (GPU-5) reaches 99.63% of BF16’s aggregate score of 8 benchmarks, being the most accu
example: https://huggingface.co/audnai/penclaw-GLM-5.3-abliterated-for-offensive-cyber. edit: good context for why this was done here https://www.reddit.com/r/LocalLLaMA/comments/1whbvpx/comment/pa16g