Search: “r-localllama”

Hey all! I'm finally doing some cool stuff with my 'thinking heater' (h/t u/-TV-Stand-). I'm still experimenting with GLM-5.2 (in anticipation of 5.3 coming tomorrow, I hope!) and things are very cool

Official Announcement: A First Look at Horus Cyber Nano 1.0 We are officially unveiling the first look at Horus Cyber Nano 1.0. This announcement presents the benchmark results of Horus Cyber Nano 1.0

I’ve been working on a small project called gemma4.c. The idea is pretty simple: you can download a modern language model, compile one 700-line C file, and have it generate text on an ordinary CPU. Th

Sharing my harness for running local LLMs that I built using Qwen 3.x 27B (> 90% locally built) under my supervision - not vibe-coded. Its free, no telemetry, and open-source (AGPL). Works on Windows,

Does anyone know of a decent reverse engineering harness/workspace setup for local models that I can just point the model at and have it go to work until it's reversed most if not all of the functions

HuggingFace releases microduck a 10 inch open-source biped with 15 actuators and sensors (camera, speaker, LiDAR, NFC, bluetooth, wifi, ...) that you train yourself with reinforcement learning, $400.

Since Qwen's dropped the Qwen4Exp architecture bomb that focus on offloading parameters to n-gram instead of pure mixture of experts, I dug into this and learned quite a lot. Here's the summary. Expec

Ever since Qwen 3.8 Flash Next dropped, there's a misconception going around that N-gram tables will let people run 1T+ parameter models on a single server with 980B parameters offloaded to SSD. I'm h

https://www.businessinsider.com/nvidia-in-talks-to-buy-hugging-face-13-billion-dollars-2026-8 Edit: The Information is reporting that the deal is done: Nvidia Agrees to Buy Open Source Model Repositor

I was using UD-Q3_K_XL until now with more than 140000 context. Quality wise it's very good, very few erroneous tool calls. Then I saw many others here reporting good results with IQ3_XXS, so I gave i

specs hardware: M4 Max 128GB Studio inference engine: oMLX & lllama.cpp insights it still very early, so had to disable oMLX K/V caching, qwen4_exp architectureis not yet supported + the obvious n-gra

I hit my token limit three times a day on my max subscription - got sick of that and designed this MCP setup to shift some of the coding load to my local Qwen3.8-27B model. I've been iterating on it n

Run it locally with one command (downloads the model, opens a small web UI): uvx --from sopro soprotts serve A year ago I posted Sopro V1 on this subreddit, a small TTS I trained for ~$250 as a side p

A lot of people are skeptical Nvidia will keep huggingface intact now that they will buy huggingface. There's a lot of doom and gloom about not having any alternatives, removing nsfw models, saying th

Hi r/LocalLLaMA ! We’re Apodex, the team behind Apodex 1.1, our new model family built to scale agentic intelligence for complex work. We’re excited to be here and answer your questions directly. Apod

A sticky popped up 'Hey, AMA today'. At first I thought I missed something, but I didn't see a single mention of it here so far, aside from having never heard of it. So, as the posting was devoid of i

With this move Nvidia is not only acquiring the HuggingFace platform, but they might also effectively acquire the copyright to the `llama.cpp` project, together with the entire team behind it. In Febr

Forgive the typos and rambling - human actually wrote this post 😂 I've had a Strix Halo board for about a year and been playing around with it for various projects when it's not just being a beefy lin

https://preview.redd.it/kbsqh6f7molh1.png?width=730&format=png&auto=webp&s=068dbea9a50be634a369d54d8b27b781d020fab3 My experience with Qwen 3.8 for agentic tasks has been phenomenal but I personally f

I wanted to see just how capable Qwen3.8-27b is locally. I have a RTX 4090 and 96GB of RAM but the Q4 comfortably fits in the GPU with plenty of context, the few times I needed more than 130k context

Upgraded to a 5070 Ti so I could run Qwen 3.8 27B, which works perfectly, but didn't want to let the old 4070 Ti go to waste. The cards would touch if I put them both in the PC and I knew the heat wou

The news about Qwen 3.8 Flash Next is the first I'm reading about n-gram tables. I may be completely misunderstanding how they work but it seems they could open the door for 1T+ parameter models to be

*part 2 of an earlier post: previous quant comparison with voxel island creation this time I rented three rtx pro 6000 96gb, on each one I launched a qwen 3.8 27b quant and gave them 4 identical promp

https://x.com/romanchernin/status/2092488160680751437?s=20 - Multimodal (Vision) - 1M Tokens Context Window - DeepSWE ~63% EDIT: THE ORIGINAL POST IS GONE Roman Chernin @romanchernin The mysterious Ox

The Artist: Qwen3.8-27B-UD-Q3K_XL, q8_0 caches, xhigh, temp 1.0, image-min-tokens 1024, froggeric template) I was screwing around with different Qwen3.8-27B quants and thought of this very simplistic

We're releasing a fully quantized NVFP4 version of Qwen3.8-27B. The checkpoint was trained using quantization-aware distillation (QAD) with QUASAR, our new QAT algorithm. We used the original BF16 mod

Hi all, I'm looking for the best model for a hobby project and trying to make sense of the various data I came across. I know benchmarks do not often translate to the real world, especially to your pa

Hugging Face is exploring sale of the business valued at around $13 billion dollars. Actually I don't think we have any other repo source. Which has the mix of model weights, datasets and Spaces. Kagg

I think that mall models between 10B and 40B are the real gamechanger. These models will be the ones that will make the AI buble pop and big companies like Anthropic and OpenAi go to hell. We are curr

I ended up fine tuning a mistral 7b to outperform our costly foundational model and saved $300k. I previously thought that fine tuning was pointless (it's definitely not) and that all these problems c

Hi everyone, it's been a while since I posted so here's an update on what the Lemonade community has been up to this summer. Our overall mission is to enable local AI builders with everything they nee

I was bored and handwrote a tiny 100-line bash script to let an agent search for and read articles from an offline wikipedia archive during a regular chat. It's not particularly useful, but it's defin

Megathread for discussing the release of GLM-5.3-Flash. Quants Fine-Tunes & Abliterations Chat Templates Inference Server Support & Configuration Experiences, Benchmarks & Model Comparisons We'll try

Megathread for discussing the (impending) release of Qwen 3.8 Flash Next. Quants Fine-Tunes & Abliterations Chat Templates Inference Server Support & Configuration Experiences, Benchmarks & Model Comp

So here's the thing, almost everyone use NVIDIA to run their LLMs, we also do the same, a lot of people we've met use like RTX PRO 6000 or even H100, B300 It seems like everyone eyes is looking at NVI

In the past, I have use llama.cpp, but I read that the exl3 quantization format should give better precision, so I have tried exllamav3/tabbyAPI. It was able to write the shown simple HTML game withou

Yet another vLLM fork thread here, but this time its for older INT8-centric hardware. This is a complete INT8 serving stack for Qwen3.8 27B based on vLLM, AITER, and a 27B GPTQ INT8 quant w/ DFlash2.

https://preview.redd.it/i8rjx0ar5mlh1.png?width=2160&format=png&auto=webp&s=c2588bc7b2519ea71b176ca73faf566dfc585496 I wanted to see whether a heavily quantized 27B model running entirely on an older

Hello, I've been self-hosting LLMs on various budget hardware for a while (6x RTX 3060 12 GB, Intel Arc Pro B60 24 GB, RX 9070 XT, etc). Over the last few months, I wrote about it in 4 articles: Gener

Benchmarked qwen3.8 xhigh, medium and muse glimmer. Xhigh effort mode with qwen3.8 took almost 30hrs. (And still failed on 16 cases because of the 32K output token limit) Medium effort mode and muse g

we made vision mlx quants of qwen3.8 27b (9 builds from 8bit at 29.5 GB down to 3.23bpw DWQ at 11.8 GB) and compared them against other community vision mlx quants from hf (we only compared vision bui

I ran 11 uncensored variants of Gemma 4 12B that I grabbed from huggingface, sorting by downloads. 10 full abliterations plus 2 LoRA adapters which were requested to be added in the comparison, agains

With hopes of a Qwen3.8-35B-A3B release now mostly dashed, many people including myself are looking at fine-tunes and other variants of Qwen3.6-35B-A3B to run on VRAM-limited hardware. I decided to tr

.. didn't llama.cpp aka ggml get acquired by Huggingface not too long ago? How would this sale affect llama.cpp and ggml? What are possible risks, and did ggerganov share his opinions or potential nex

'A 12-core GPU, also with two more cores than before, now includes Neural Accelerators in each core for the first time on Mac mini, resulting in up to 4x faster AI performance and 2x faster graphics t

TL:DR: power management was doing auto-suspend and dumping model weights into RAM. Models weights 18gb+ RAM 16gb = no Bueno immediate crash. amdgpu.runpm=0 in your grub disables runtime power manageme