Knowledge catalogue
TL;DR: switching KV cache to f16 may give a boost in speed if using MTP and ngrams. I have a self-built 'AI mega-cluster' with 2x P40s on a cheap Chinese motherboard and a Xeon CPU (around $1,100 to b
Was considering choosing a 4xRadeon AI Pro R9700 build for a office AI server. Curious to know what results you guys are getting? Mainly considering DeepSeek V4 flash and Qwen 3.8 Flash with offloadin
This comparison was requested by a few people, and certainly we were all eager to see the final results. The comparison had taken 11 days and the GPU was crunching numbers for ~167 hours. We've been c
Hi everybody, I am in a tricky situation. My grandfather + grandmother live in a remote village in asia, and were visiting us in the city. My grandfather has absolutely fallen in love with AI, and it
So after all my work, yeah, Raymond did it better, so I ported his work over, extended it turboX, extended it multiple other models (he had only Qwen models), and benchmarked the crap out of it to mak
While we see for frontier models similar scores among famous coding benchmarks, across: DeepSWE, Terminal-Bench, LiveCodeBench, Code-Arena ELO. Here are in my opinion some next level benchmarks that r
I'm using DS-V4-Flash-Vision with Q8_K_XL quantization locally as my everyday engine, and for some time now I've been doing a lot of comparisons with Qwen3.8-Flash-Next, also with Q8_K_XL quantization
Hello everyone. I felt current LLM benchmark harnesses hand you headline numbers but offer no tooling to see how models actually answered each question (they dump everything to a JSONL or Parquet file
I only use free tiers of these large models to offset compute while my own system runs and for 'different' points of view, since what pops ups suggestions seems to vary a lot sometimes, even when buil
https://huggingface.co/XHToken/Spark-X2.5-4B-GGUF https://huggingface.co/XHToken/Spark-X2.5-1.7B-GGUF from XHToken: We are introducing Spark-X2.5-4B and Spark-X2.5-1.7B, two compact, general-purpose l
When I first started working in scientific machine learning, I understood the physics much better than the coding. Every time I wanted to try a new physics-informed neural network problem, I had to st
Hi, I've been wanting to run my own local LLM for some time and I finally saved enough to get a budget GPU. I can spend 700 at most, and I'm looking for a GPU that can run quantized 30B~ parameter mod
Right, so this is going to be embarrassing but it's presumably something we've all been through at one point or another, and I guess this is my first time resolving something like this in the way that
I feel that reproducibility is now a lost cause in machine learning research for three reasons: Many research is moving towards the physical AI territory, where you need expensive hardwares or even en
Hello, I'm a bit obsessed with cache management on local LLMs. For the last few days I've been working on cache management on vLLM with my 2x DGX Spark and DeepSeek v4 Flash 0731. I felt something was
You need Blender 5.x with MCP extension. I use pi so I also need: pi install npm:pi-mcp-adapter and then: $ cat .mcp.json { 'mcpServers': { 'blender': { 'command': '/home/jacek/.local/bin/uvx', 'args'
https://village-sim-one.vercel.app/ - 16GB VRAM RTX 5070 Ti, fully offloaded - Vision on CPU - Windows, not headless - beellama.cpp - latest version with the kvarn performance enhancements making it a
Hi everyone I'm running four rtx 4090, 64GB ram, on a threadripper pro motherboard so all PCIe x16 ports, as a homelab machine for coding. I was migrating from vllm + qwen 3.8 27B (fp8+256k kv cache)
Hugging face has over 300 versions of qwen3.8-27b. I have no idea how to identify the beat model to download and use. Should I sort by “most likes” or “most downloads”? Is there any sort of other guid
I ran a side-by-side ML text-processing and model-training workflow using Fable 5.1 vs. Astra (both on xhigh), and the results could not have been more different. Warning, long post. TL;DR -- Astra co
I just spent the last 2 hours of my life on a Friday night debugging a strange error in a prod CLI app. EF core was receive a readonlyspan during a Contains query. Normally, this query converted to a
Time for another update! We have been busy and managed to improve the gains substantially (mostly from exploring existing llama cpp PRs and adopting relevant things). Among other things the README.md
A researcher has reported a jailbreak of GPT-6 Astra within a day after release. The attack is described as combination of TIP (Task-in-Prompt) attack from ACL 2025 paper with four other unnamed techn
I have a 5070 Ti and two 5060 Ti (all 16Gb cards). I planned to add another 5070 Ti giving me two pairs of 32Gb each but the NVidia prices have just jumped by 25% where I am and I've found a 3090 Foun
I got a lot of ads for meshy recently and went to try it with hilariously bad results. It apparently can't do anything but decorative figurines. I wanted a body shell for an rc car and it just couldn'
Anyone who has a 3D printer and get use of it finds it incredibly useful for those odd jobs around the house, a missing bracket, a cable router, steam deck holder and so on. In the past if I was missi
OK, you guys want to start with local vibecoding but don't know how. In the discussions, you recommended that I use Godot. I downloaded it without knowing what it was. I was expecting a library, but i
I've updated my auto_demo_scener project with Ninfer support and a “rewrite based on video” feature that I thought you might find interesting. The project is basically an endless demoscene machine. A
I just want to use paperless-ai and be able to ask questions relative to it. Bonus points if I could use it with home assistant but that's not the focus. I just can't see needing a 32 GB VRAM GPU for
Hiya, NInfer is amazng for Qwen, but lacking for real-world-use. As adoption of issues/pr's was not really what I needed, I created a fork and hit it for this week with 3 concurrent claude code sessio
I've been running Qwen3.8-27B as a local inference server for a production content intelligence pipeline (HVAC industry stuff, lots of long-context retrieval and structured extraction). I have been wa
Otaku is an LLM frontend, primarily designed for roleplay, an alternative to SillyTavern and the like. However, It also works for general-purpose chat with local backends (including Ollama) or cloud m
Used qwen3.8-27b in Opencode to make this silly mini-game because I'm not sober: ``` We are going to play a game, it will be the Wikipedia game. The Wikipedia game has the following rules: You will ha
First, I'd like to thank the Qwen and Unsloth teams for the Qwen3.8 27b UD Q4_K_XL. Fits the poor 24GB of 3090 VRAM with 100k context at Q8 and works phenomenally well! Imho if theres anything that ca
Ever since u/jfowers_amd has asked me to help with the Lemonade project (and provided some hardware to test on), I've been trying my best to optimize llama.cpp for AMD setups. This has led me in some
I was running into a lot of posts that praised both the Fixed template and the Sharp template in comparison to the stock one, so I put them to the test. It's not as extensive as it should be for a pap
Qwen3.8-Flash-Next (UD-Q4_K_XL) on a single RTX 3090 24GB + 128GB DDR4 — is this config optimal? Hardware CPU: Intel Core i5-12600K RAM: 128 GB DDR4 @ 3600 MHz GPU: NVIDIA RTX 3090, 24 GB VRAM OS: Win
Hi, I am thinking about putting some money / time into my local ai learning path and 'cleared' my attic where I found the following hardware. 3 x NUC11 (Core i5 1145G7 / 2,6 GHz, 64 GB RAM DDR4 2666MH
I've been trying the latest models from the frontier labs and honestly, after extensive testing I can not tell the difference between the best open source options. I think the differences are now marg
NVIDIA ships two neural networks inside DLSS: the DLSS 5 renderer that makes game frames look more photoreal, and the frame generator. Both only run on RTX cards inside games that call DLSS. I pulled
I see a new one being launched every few days... How do these new harnesses compare to claude code, pi etc. has anyone switched from these? which harness to prefer and why submitted by /u/Background-J
I built a denser GGUF format for ternary models: Q2_B3 / “B3S” If you're running a ternary model like BitNet-b1.58 or Ternary-Bonsai, the weights are already restricted to -1, 0, or +1 times a block s
I always heard that cryptocurrency mining can damage a GPU ( it's maybe wrong) so how much long term inference is damaging a GPU? I mean servers are made for 25/7 operation... Gaming GPUs not so sure?
Hey everyone, been a while! https://huggingface.co/TheDrummer/Artemis-31B-v1.1 https://huggingface.co/TheDrummer/Artemis-31B-v1 A few months ago, Gemma graced us with models that served as a much need
A month ago I posted about Gemma sometimes solving the reasoning problems inside my translation data instead of translating them. A few people suggested two very fixes, which is to use a proper transl
How good/bad is it against comparable MoEs the same size? How does it compare against Qwen 3.6 35BA3B? Since we don't have 3.8 35B this seems like an upgrade if we look at some benchmarks like termina
Let's say I have 2x 16Gb GPUs and I want to run Qwen3.8 27B. Monitor is ran by the integrated GPU so both 16Gb GPUs are almost fully free. I load the UD-Q4_K_S on one card at 15.4Gb. I then load the c
I am getting into developing a RAG setup, for getting information out of existing documents, new document ingestion, web searches, and good visuals. I am planning to use it for, alongside the regular
We all want more tokens per second but I keep seeing confusion on what to expect for given hardware. For the decoding phase (TG/s) to produce one token all the model weights and KV cache needs to be r
After Qwen3.8 27B came out, I decided to benchmark the models that could fit in my GPU (RTX 5080) on my actual code (C code), the results were not completely unexpected but some quants were definitely
This epyc server I am using twelve cards with 64 GB memory, plus 256GB ram. Looking at the most capable models in open source, GLM 5.3 seems to be the only option, but with Astra releasing it will lik
It performs well across visual perception, STEM reasoning, document intelligence, multimodal agent tasks, frontend coding, and medical report interpretation. submitted by /u/niacolhealth [link] [comme
This is nothing impressive but, i had so much fun i wanted to share my experience with this model. (yes this post is written by human) I made an FPS with local Q4_K_XL 3.8 Flash Next (256k context) (i
You know that thing where you just throw a task at a frontier model and not have to supervise it worrying of it going off course? Qwen3.8-27b has officially gotten me to that point for local work. He
I've got an refurb Dell R740 running Proxmox that I put a Tesla T4 in, mainly to run some CTC local transcription work, but thought it would be fun to try DS4 when it came out, and it was appalling at
Claude Code is absolutely horrendous at managing docx XML structures and it's obscene that the best these frontier companies can do are skills. Skills are prone to breaking, burn tokens (hello busines
Being a person obsessed with testing new models that come out, times are really insane for me. Tested different kinds of TTS and voice cloning models but none of them gets it right in terms of emotion
This is a follow-up to my post from yesterday (17 -> 25-29 t/s with the expert cache PR). Same box: 2x RTX 3090 on PCIe 3.0, dual Xeon E5-2696 v4, 188 GB DDR4-2133 LRDIMM, llama.cpp, full 261k context
I was using: weights https://huggingface.co/RadixArk/Qwen3.8-Flash-Next-NVFP4 with the optimized SGLANG (patched) from https://old.reddit.com/r/BlackwellPerformance/comments/1w04xb7/qwen38_flashnext_o
I'm one of the developers. We said in August it would go open source in September and it did last night. MIT or Apache-2.0, pick one. The repo you see is our internal repo, kernels included, so from n