Search: “r-localllama”

I find this new model at HF: 'Built for demanding work. A 262 144-token context window, adjustable reasoning effort, tool calling, and text, image and video understanding. Architecture Agnes-3.0-Flash

Q2 is there and Q4 is uploading as I type. Has his github been updated yet? How do you run this? https://huggingface.co/antirez/deepseek-v4.1-flash-gguf/tree/main submitted by /u/Queasy_Asparagus69 [l

Hey everyone! I'm curious to hear from people that use a combination of cloud-based frontier models and local ones for development. I'm planning to set something similar up and wanted to hear about ac

Blog Post : Per-tensor layout maps for GGUF quantization Reddit thread : New tensor type layouts for my GGUF uploads EDIT : Model card has updated things such as Graph, table, text, etc., submitted by

I have a full M5 Pro Macbook Pro with 48GB of RAM, I'm just getting into this local space. Like many of you, the costs of using frontier/cloud models is getting kinda silly. I don't have any insane ne

So, uh... the popularity of so-called humanlike Qwen (currently on top in this sub) made me realize just how clueless the general public is about the models they have. You'd be shocked but you don't n

Kimi routed some PLA requests to Claude for distillation purposes without warning the PLA users. There is rumor that 16 Moonshot employees were arrested for this leak. submitted by /u/Ok_Warning2146 [

Hey everyone — I’m building TensorSharp, an open-source LLM inference engine. Here are the latest DeepSeek V4.1 Flash GGUF results using its native ggml_cuda backend. Setup: 8× NVIDIA A40, layer split

First off, I know that GLM, Qwen, and DeepSeek absolutely dominate in terms of SOTA Open Source models, and that’s what I use in my personal projects and for school, however, I’m also responsible for

For many, Vram is precious, I see many people recommend using --no-mmproj-offload to save gpu vram but it is painfully slow. Especially if you are using it with agentic coding. If possible, add that s

I have enabled the vision for the CIRU Strix UL4 quant of Qwen 3.8 flash next (others quants likely perform very similar) and tried it on a few things, then wanted to show my partner how great it work

https://github.com/Sayitobar/chat_llm_cookbook This is my personal project that took several months. I wanted to see whether a 2B small local model could simulate a six-person group chat trained & ran

I've seen a lot of Super Mario showcases, but never a mashup or derivative work. So I took a Super Mario generation prompt from ChopSticksPlease's Post here - and added a Matrix twist - and found myse

Another new model dropped in the course of this week that is well deployable on consumer hardware: Nex N2.5 Mini I went with the recommended settings for the best generation quality and ran a few benc

Hi! I run both models on MTPLX on my m5 max, and since I have 128GB of ram I run the q8 27b. I think MTPLX only lets me run 'optimized for speed' which it says is a dynamic q4 with 8 bit attention. Bo

Hola all. Do you guys mind sharing your LLama.cpp config and system setup details for Qwen3.8 Flash Next? Model's quite big and tryining many combinations of llama.cpp options takes lots of time, so l

As you all know, Qwen3.8 Flash Next on mainline llama.cpp is still in a pretty experimental stage, but a lot of community forks are trying to get it to work better. There's also a closed-source soluti

AuK-Flash: Fast 4-Step Speech Generation and Editing arXiv : https://arxiv.org/abs/2609.08936 Full Paper : https://arxiv.org/pdf/2609.08936 GitHub : https://github.com/Tencent-Hunyuan/AuK Project : ht

In my opinion the perfect model(s) will be around 20B-32B and A2B-A4B. This is the sweet spot on which companies should focus on. Qwen 3.8, Gemma E2B/E4B and Ling 3.0 proved it. They are still imperfe

https://huggingface.co/HermiHg/Qwen3.8-27B-DFlash2-Q2_K_S-MIX-GGUF I used this draft model with https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF with the IQ3_XXS with 128k context and I saw

I have been battling with my Qwen3.8:27b setup on my rtx 5080 16gb. I am using llama.cpp to run a nvfp4 version of qwen3.8:27b llama-b10699-bin-win-cuda-13.3-x64llama-server.exe -hf williamliao/Qwen3.

While browsing a benchmark list site, I spotted a recently published 33B parameter model which claimed to beat Qwen3.8 27b on the ArtificialAnalysis (AA) intelligence index. I was obviously excited. B

Planning to build a PC mainly for local LLMs/coding agents. I keep seeing 3090 + Qwen 3.8 Flash Next benchmarks, but could not find enough info for the 7900 XTX 24GB. 3090s are hard to find where I li

Hi! I've been following this community for quite a while and have difficulty figuring out what to put on my 3080 12gb - I know Qwen 3.6 35B 3A was the go-to choice when it first came out, but I'm curi

We have SWE-bench, Terminal-Bench, OSWorld, BrowseComp, etc. But I haven’t seen a good apples-to-apples benchmark for platforms like OpenAI Agents, Anthropic’s agent stack, AWS AgentCore, Google’s age

Hi everyone, I was interested in learning LoRA fine-tuning, and ended up building CodeFinetuner over the past few months, a full pipeline that fine-tunes a small code autocomplete model (e.g. Qwen2.5-

I wanted to see how my fully local home voice assistant compared to the latest GPT Live, so I tested it using the same conversation used in their 'Improved Intelligence' demo. In this video they ask t

A huge favor to ask llama maintainers - please implement this feature. Even with one 3090 card there will be tangible improvements in decode speed on MOE models with moderate number of active paramete

Hello Guys, Just sharing this harness I've created, this runs entirely on chromium based browser. No installation is required, it has its limitations and by no means is supposed to replace better agen

I just watched a YouTube from Luke’s Dev Lab where he literally just plugged a RTX 2000 ADA Into the side of the Zima Board 2’s PCIE socket and it just friggin worked and had great token speed despite

Hey gang, im wondering if you in theory could use ngrams as seen with Qwen 3.8 Flash or DS4.1 in order to dynamically train the model? Normally the ngram embeddings behave similar to a lookup table of

I've been out of the loop for some time. Is there already an official way to offload ngram to RAM or SSD in something like Unsloth Studio? Interested in running Qwen3.8-Flash-Next on 72GB VRAM, but na

I haven't seen this mentioned yet, so I thought it deserves a post. I was trying out OpenWhispr when this model came up as the recommendation. So I don't have personal experience yet, but it's suppose

I made this because I was getting genuinely annoyed at trying to have a normal conversation with LLMs. Even with prompting and various tricks, most models I've tried still have this 'AI assistant' vib

I can't stand kv cache quantization. Even at q8_0, I can feel the difference. But realistically, when running Qwen3.8-27B-UD-Q4_K_XL on my 32 GB GPU, I only have room for ~170k tokens (with mtp and mm

I’ve been building 'Spomin', a router that replaces context with summaries directly in the KV cache. The goal is to keep long running sessions going without repeatedly stopping for full compaction and

Some people says terminal bench reflects model intelligence better than the intelligent index. From the look of it, the ranking does seem to reflect how people feel about the open and closed models. F

Astra's launch has produced a strange discourse. The reporting that broke the story framed the model's use of recurrent depth primarily as a safety regression, because it means the model reveals less

i've been thinking about this more after trying different agent setups. the model isn't the only thing that determines how well an agent performs. The harness around the model matters a lot too. With

I wish I could spend $15k on my own homelab hardware, but no this is for work lol. Like the title says, we're looking to run DSV4 Flash (and similar tier models) locally at good speeds, both for token

Looking for tools to build/run an engine to do STT-LLM-TTS basically. I know how to run all of them separately, and if need be I can build one myself, but I was curious if there is already an open sou

I'm currently building my system around 3060s, but I might be able to get a 4060 for a nice deal. At first it seemed like a no brainer, but turns out the 4060 has lower memory bandwidth. In a system t

Spoke with many providers who lurk in this sub, plus met folks who work in inference. For a 10k monthly revenue, a provider was able to retain only 200 dollars in profit due to GPU costs. Their custom

Like many of you, I have seen many posts and tweets in the last weeks complaining about Artificial Analysis being 'broken', 'meaningless', and 'bought out.' People who say this have done no research a

OpenAI has decided to fully shut down a protein design project I'm working on for a client. Needless to say, open weight models are the only way forward. submitted by /u/Terminator857 [link] [comments

Title says it all. https://github.com/gjabdelnoor/Day1DeepseekV4.1-CPU The goal is pretty simple, I like having infinite slow tokens from the bioinformatics machine in the lab to run overnight or over