Search: “context”

Some people here say MiMo-V2.6 is bad with tools and are going back to GLM-5.3-Flash. I spent today running MiMo-V2.6-Flash-RL as the backend for an agent harness, on 2× DGX Spark with vLLM, using the

I've been thinking a lot about the idea of how a human mind is similar to a digital neural network. Seeing videos of the fruit fly brain scan, and the crazy things people have done with it probably co

Sharing my successful experience here. My setup: RTX 5060 Ti 16GB (x16 PCIe slot) RTX 3060 12GB (x4 PCIe slot) RAM: 32GB CPU: Ryzen 5 5600 OS: Linux So I have a total of 28GB. Average tk/s: 50, and th

I am trying to learn more about long-term memory and how people are handling that nowadays. Historically, I have used a mixture of paid cloud AI services (ChatGPT, Gemini, OpenRouter API) and my own l

TL;DR: I run a mismatched Tesla V100-PCIE pair—one 16 GB card and one 32 GB card, 48 GB total—in a Proxmox/LXC-based local-inference lab. The practical winner so far is a recent CUDA build of llama.cp

just saw some benchmarks on the omlx website for the m5 ultra (don’t know how official they are but they seem reasonable): Link For Qwen 3.8 27B q4 it gets 50 tok/s th and 1800 tok/s pp at8k context a

I find another new model at HF: https://huggingface.co/XingChen-AGI/Xing4.0-29B-A4B 'Xing4.0-29B-A4B is a next-generation large language model in the Xing series (formerly TeleChat), developed by Chin

Hey experts! I genuinely have this question and would love a general consensus from people actually using these models. LLMs have advanced a lot in benchmarks and in practical usage. Even GPT-4o and G

example: https://huggingface.co/audnai/penclaw-GLM-5.3-abliterated-for-offensive-cyber. edit: good context for why this was done here https://www.reddit.com/r/LocalLLaMA/comments/1whbvpx/comment/pa16g

I have tried using qwen 3.6 and 3.8 q5 64k context and I'm just not getting the experience everyone here seems to get. I'm using open code and it will loop, forget what it's doing, and mess up the ui.

Hello Reddit. Posting this for fun. I thought it was a lonely and silly journey to set up Qwen 3.8 Next on a system that doesn't really run it properly—it was a challenge that might help the community

Want to help me with a tiny ChatGPT experiment? I stumbled onto something interesting after seeing how differently ChatGPT users got responses to the exact same underspecified prompt, and now I’m curi

I've been keeping journals on and off for more than 10 years. There were stretches when I wrote pretty regularly, then I'd stop for a few months. Usually I'd write down what happened, what I did, mayb

Another new model dropped in the course of this week that is well deployable on consumer hardware: Nex N2.5 Mini I went with the recommended settings for the best generation quality and ran a few benc

Here we go again, DeepSeek is back again with a new model V4-1 Flash A multimodal Mixture-of-Experts (MoE) model with 552B backbone parameters and support for contexts of up to one million tokens Mark

Hey y'all! We've released a new model in our lineup: GigaChat-3.5 Reasoning. It's a 432B-A28B MoE with Gated DeltaNet for long-context efficiency. We trained domain experts (code, math, general, etc.)

I initially saw CED as just an efficiency improvement, but the more I read about it, the more it feels like an inference architecture leap. The encoder/decoder split has some pretty interesting implic

mentria.ai is a browser inference engine I've been building solo, from scratch in WebGPU/WGSL. This week it crossed a milestone I had been chasing for a while: a 27B one-bit model answering at up to 3

Hi, I'm wondering what settings you are using in order to run Qwen3.8-Flash-Next on your devices? I'm especially interested in setups with 96GB VRAM. I'm not quite sure if llama.cpp does offload the e

Up front: I'm one of the people building the page-perception layer used here. We started by testing small local models. The result turned out to be more interesting than the original test. 12 small mo

Quick setup on linux: STEP 0: install/download llama.cpp, Freecad, your favourite gguf model - possibly with multimedia image reading capabilities (I've used Qwen3.8-27B-UD-Q4_K_M and relative mmproj-

I got 2x 20GB RTX 3080s + 128GB of DDR4 2666hz RAM (only 4 of 6 channels populated) + a Xeon 6148 I've always been a llama.cpp person and I've been running Unsloth's Q4_K_XL quant of Qwen 3.8 Flash Ne

How it works. The model being tested is given a server capable of running it's weights and full context. That server is placed in a median priced apartment. The AI is given a bank account with for ren

I've hit again a point where me as a developer have to take a break from all this slop shit. Im a Developer for 13+ years and i loved it. But i fell for the slop trap. First it started with copilot an

I keep seeing these massive ultimate chatgpt prompt posts all over social media, like we’re still using Chat and other LLMs the same way we did when they first came out. Meanwhile the more I use AI, t

Hi, I've been wanting to run my own local LLM for some time and I finally saved enough to get a budget GPU. I can spend 700 at most, and I'm looking for a GPU that can run quantized 30B~ parameter mod

Ever since u/jfowers_amd has asked me to help with the Lemonade project (and provided some hardware to test on), I've been trying my best to optimize llama.cpp for AMD setups. This has led me in some

After Qwen3.8 27B came out, I decided to benchmark the models that could fit in my GPU (RTX 5080) on my actual code (C code), the results were not completely unexpected but some quants were definitely

Vellium is an open-source, local-first desktop app for AI chat, character roleplay and long-form writing. Recent updates have focused mostly on making local voice and model setups easier to use. Live

TL;DR: this paper proposes a method to fix hallucination rates to very low levels or zero by disabling neurons which contribute to hallucination. This discovery has been out for a while now, but it ha

The intelligence in my context is an aggregate index, I called the Agentic Coding Index, across most relevant agentic coding benchmarks: SWE-bench Pro, DeepSWE v1.1, Terminal-Bench (v4, v3, v2.1), Cod

Running on a M1 Max 64 GB: - SSD streaming for tensors - SSD streaming for engrams - SSD streaming for MTP How is it possible? * Custom Q4 quant: benchmarked all metal tensors then picked and spliced

If you own 4xR9700 and were waiting for the model to make them shine, then I have some good news for you! It's running at 80-120 tokens/second for generation and 12k token/second prefill for a single

Post rédigé avec l'aide de GLM5.3 flash Contexte : formateur professionnel d'adultes en France, j'utilise des agents LLM au quotidien et le français qu'ils produisent est mon critère n°1, avant les be

I hit my token limit three times a day on my max subscription - got sick of that and designed this MCP setup to shift some of the coding load to my local Qwen3.8-27B model. I've been iterating on it n

I've been building a small tool opensource project that sits between applications and local LLM servers (starting with Ollama) to diagnose silent inference failures instead of just showing logs. Thing

https://x.com/romanchernin/status/2092488160680751437?s=20 - Multimodal (Vision) - 1M Tokens Context Window - DeepSWE ~63% EDIT: THE ORIGINAL POST IS GONE Roman Chernin @romanchernin The mysterious Ox

we made vision mlx quants of qwen3.8 27b (9 builds from 8bit at 29.5 GB down to 3.23bpw DWQ at 11.8 GB) and compared them against other community vision mlx quants from hf (we only compared vision bui

One of the things I like most about Ollama is how low it makes the barrier to experimenting with local models. The downside is that once you move away from high-end hardware, finding the right informa

Granite-4.2-30B is the flagship reasoning model in the Granite 4.2 family. It delivers the strongest performance across reasoning-intensive tasks by leveraging built-in <think>...</think> chain-of-tho

I’ve been going back and forth on this for a week and I can’t settle it, so I’m hoping someone here has hands-on numbers. The two configs (German prices, dealer quote, incl. VAT): Config Price Mac Stu

good morning, has anyone already used qwen 3.8 with the tesla v100 or sxm2 or pcie? any information is useful to me context token/s thanks for the help. submitted by /u/Full-Chicken2110 [link] [commen

I ran the Aider benchmark on Qwen 3.8 27B FP8 with FP8 KV cache 256K context vLLM. The score: 72.9 This matches Gemini 2.5 Pro from 2025-04-12 which also scored 72.9. Beats Claude Opus 4 from 2025-05-

With vision enabled I am able to run at 150k context on a single RTX 3090 and the results are just amazing. I was even able to write a gmail plugin for DeepSeek harness with locally hosted Qwen 3.8 27

WARNING: I think I made a mistake with omlx, and its results below seem to be without any kind of acceleration. I am rerunning them with Lightning MTP and ANE support for prefill. Will update the post

Hello, So I've been trying lots of combinations in that never-ending landscape of options and settings. I wanted a proper quant of 3.8 27B running as fast as possible on my 5090 at 400W, with vision a