Search: “hardware”

I released KaozKit this week, a Swift library for running LLM agents written in JavaScript inside a Mac app. Built from the start to keep everything local. Providers: MLX with Hugging Face model manag

Achieved steady 15 tokens/second output and 100-120 promp processing per second with 12GB GPU (RTX 5070 SFF) with Qwen3.8-Flash-Next-GSQ-RCO-GGUF at 3 bpw (IQ3_XXS which maches BF16 on AIME25). Around

I wanted to test what Qwen 3.8 27B running locally on a single 5090 could do in Row-Bot. I gave it just one prompt: 'I want you to create an animation showing what the current local model can do. Test

Follow up post of https://www.reddit.com/r/LocalLLaMA/s/nGMyKswrch. Thanks everyone who replied. I didn't change the specs. Might be loosing some of the memory bandwidth but will scale in future if I

I've been working on something for this community (and other self-hosted communities) for almost 5 months now. Personally, I would grade it as probably the closest open source equivalent of Manus and

I liked DeepSeek V4.1 Flash as an agent model but at 16 t/s on ds4 it was painful to sit through a real turn. It seemed some redditors and m3 ultra owners appreciated my glm 53 flash optimizations, so

Hello Reddit. Posting this for fun. I thought it was a lonely and silly journey to set up Qwen 3.8 Next on a system that doesn't really run it properly—it was a challenge that might help the community

I have a full model, it's ready to train. It's ~9b parameters. 9.4b to be more exact. That includes a 1/2/3 Engram table, Moonshot's AttnRes modeling, and RoPE / NoPE layering at 3:1 as more or less v

First off, I know that GLM, Qwen, and DeepSeek absolutely dominate in terms of SOTA Open Source models, and that’s what I use in my personal projects and for school, however, I’m also responsible for

I have enabled the vision for the CIRU Strix UL4 quant of Qwen 3.8 flash next (others quants likely perform very similar) and tried it on a few things, then wanted to show my partner how great it work

Hola all. Do you guys mind sharing your LLama.cpp config and system setup details for Qwen3.8 Flash Next? Model's quite big and tryining many combinations of llama.cpp options takes lots of time, so l

Tested on Laptop with RTX 5060 8gb vram and 16gb ram Claude code - autocompact = false Ollama - Flash attention = 1, kv cache = q8_0 Gemma4 12B 64K - https://ollama.com/SetneufPT/Gemma4-12B-IT-QAT_Q4_

Hi everyone, I was interested in learning LoRA fine-tuning, and ended up building CodeFinetuner over the past few months, a full pipeline that fine-tunes a small code autocomplete model (e.g. Qwen2.5-

I’ve been building 'Spomin', a router that replaces context with summaries directly in the KV cache. The goal is to keep long running sessions going without repeatedly stopping for full compaction and

Both good and bad things have come from a subreddit that was lot more niche than for example r/flashlight rapidly transforming into the largest online forum about an increasingly core part of the infr

Part 4 of the same box. Part 1 was 17 -> 25-29 t/s with the expert cache PR, part 2 was 37-41 t/s after switching to UD-Q4_K_XL and stacking MTP on the cache, part 3 was the top-k fallback that was so

Just a reminder for those out there itching to get into local llms - don't let FOMO or 'gear acquisition syndrom' take over. No matter the hobby, it's so easy to get stuck in a trap where we buy more

Cosmos3 INT4 T2I + I2V on Apple Silicon — code, weights and a Grok comparison GitHub - https://github.com/gtrg55/cosmos3-quant-mlx-cuda HF weights - https://huggingface.co/JuliaML/Cosmos3-Super-Text2I

Found this post:https://www.reddit.com/r/ClaudeAI/s/At9QNEbawK of someone fully developing a game through AI writing code, generating 3D models, etc. (he is using Meshy to create the 3D models). I was

I've been making a lot of comments about optimal setup for Strix Halo (gfx1150) and from my observation, 90% of our community is using offcial llama.cpp for it, which is NOT optimized for Strix Halo a

I'm the CTO of a mid-size nonprofit, and I run a local Qwen stack as a coding subagent for that work — partly on principle (a fair amount of what I handle shouldn't leave the building), partly because

Hello LocalLLaMA Fam! I built this project because I, like many of us here, enjoy local AI and have a nice setup, but still find myself reaching for Cloud-based AI due to their convenience. My models

Hey guys, After my CPU-only to 96GB VRAM test, I tested Qwen3.8-Flash-Next across llama.cpp, SGLang and FreeToken on the same workstation. This time I wanted to see what changes when you keep the hard

I wanted a quick calories counter for myself, using LLMs to evaluate the calories from pictures of meals + descriptions. I needed to pick a model so I made a quick benchmark. The setup was: - Nutritio

The LLM is limited to NPC emulation. Game state, world logic, quests, and the authored story are handled by deterministic game systems rather than the LLM. I built it this way because I wanted the fre

LayerStoRm: GLM-5.3-Flash UD-Q4_K_XL (186 GiB) at 1M context on 2× RTX 5090 + 2× RTX 5080 (96 GB VRAM total) using RAM for the pinned experts. LayerStoRm is a (still experimental) MIT-licensed continu

I've built and open-sourced ParaRNN, a PyTorch/Triton library that parallelizes the training of non-linear recurrent architectures across time using Newton-Raphson iterations mapped to associative pre

I recently ported VDN-H3 (Video DeltaNet for MiniMax H3) to Vpipe and tested it on an M5 Pro 24GB. The benefit scales with both sequence length and resolution: at 832×480, VDN-H3 starts to pull ahead

I've been tinkering with local LLMs since the beginning of the year when I had an Intel Arc B580 and 32 GB of DDR5. Curiosity got the best of me and I bought the first R9700 about half a year ago, als

TL;DR: switching KV cache to f16 may give a boost in speed if using MTP and ngrams. I have a self-built 'AI mega-cluster' with 2x P40s on a cheap Chinese motherboard and a Xeon CPU (around $1,100 to b

This comparison was requested by a few people, and certainly we were all eager to see the final results. The comparison had taken 11 days and the GPU was crunching numbers for ~167 hours. We've been c

Hi everyone I'm running four rtx 4090, 64GB ram, on a threadripper pro motherboard so all PCIe x16 ports, as a homelab machine for coding. I was migrating from vllm + qwen 3.8 27B (fp8+256k kv cache)

Hiya, NInfer is amazng for Qwen, but lacking for real-world-use. As adoption of issues/pr's was not really what I needed, I created a fork and hit it for this week with 3 concurrent claude code sessio

I've been running Qwen3.8-27B as a local inference server for a production content intelligence pipeline (HVAC industry stuff, lots of long-context retrieval and structured extraction). I have been wa

First, I'd like to thank the Qwen and Unsloth teams for the Qwen3.8 27b UD Q4_K_XL. Fits the poor 24GB of 3090 VRAM with 100k context at Q8 and works phenomenally well! Imho if theres anything that ca

I was running into a lot of posts that praised both the Fixed template and the Sharp template in comparison to the stock one, so I put them to the test. It's not as extensive as it should be for a pap

I've got an refurb Dell R740 running Proxmox that I put a Tesla T4 in, mainly to run some CTC local transcription work, but thought it would be fun to try DS4 when it came out, and it was appalling at

Two weeks, one rule, and hundreds of entries from nearly 50 countries. Here's who took the four titles, and a look at everyone who made the final ten! Entries opened August 20 and closed September 1.

I've tried both and been having this debate with myself for the last few days, on two Asus Ascent GX10s (effectively the same as 2x DGX Spark): DeepSeek-V4-Flash-0731 (official weights) GLM-5.3-Flash

If you are like me and use Google flow for NB2/NBP there is hope to finally ditch it. Let me start off by saying I’m not very impressed with any of the current local image edit or Image Reference mode

Quality: 81.1 → 87.7 (+8%) Speed: 35 → 29 tok/s (−16%) Runtime: 8m51s → 44m39s (5x longer) Output tokens: 18K → 78K (🤯) Noticeably better quality, but you're paying for it with tokens and time. Full b

Hey everyone, Wanted to share a short paper we just published exploring a simple but surprisingly effective optimization for sparse MoE reasoning models. The problem: MoE reasoning models (especially

This was my first serious attempt at tuning a local LLM. I started because Qwen3.8-27B IQ3 was fast on my RTX 5080 but the coding quality disappointed me, and the Q4 profiles I tried in LM Studio were

Alright, I posted that I had my agent test a bunch of different workflows for over 12 hours and got the 'Bro just wasted 12 hours of credits'. It was obvious the proof should come from the visual data

(Paid 5,849 + tax, which came out to 6,279) TL;DR - Bought this 5090 prebuilt and I want to sanity check if I made the right decision and at the right time. Hey everyone. So I want to start off by say

Hi all, I have recently been experimenting with different LLM set ups and after everyone was raving about how good Qwen 3.8 27b was, I was inspired to try and get it deploying. After some battling wit