All entries

My test is running DeepSeek v4 Flash 0731 at native size on DGX Spark machine (GB10, 128 GB unified memory). The model size is bigger than RAM, so weights will be loaded many times when running. To im

Graph: A cumulative metric of Speed x Context Size - the higher the dot - the better. Helpful for looking for the most balanced solution. The research relies on DFlash2 Q2 work by AnalogAlok: https://

Share what your favorite models are right now and why. Given the nature of the beast in evaluating VLMs (untrustworthiness of benchmarks, immature tooling, intrinsic stochasticity), please be as detai

Hi everybody! Every now and then these days, we’re seeing really huge open-weight models popping up. But since not everybody has a DGX Station at home, I’m interested in really small models. It’s incr

I just finished building my (relatively) low rent local inference machine: * Epyc 7663 * 256GB ECC DDR4-3200 * 1x RTX 5090 32GB Yeah I realize it's weird to throw a 5090 and 256GB of anything together

Standard constrained RL assumes consequences are immediate and attributable to the current action. This breaks down whenever violations are delayed and stochastic, which is most real-world settings yo

I have deleted tons and tons of older models to make space since I can't afford storage anymore. Easily 10TB... Anyways, I have been considering deleting DeepSeekV3.2 but decide to run it one more tim

A month i setup a proxmox server using spare parts like a z270 i7 7700k and 2x8 2666 but since i don't have any hardware acceleration i was planning to pass down my rx 580 8GB to my server but i heard

As per title. I'm not affiliated with the team behind this model in any way, shape or form. As a GPU poor myself (8 GB VRAM laptop + 12 GB VRAM desktop), I found Laguna to be very promising on my lapt

hello everyone. I'm training PPO variants on different multi-agent tasks from the VMAS library (Independent PPO / Graph PPO and such, see HetGPPO by Bettini et al.). I noticed that for every architect

Reposting here after sharing this on [r/MachineLearning](r/MachineLearning) a few days ago, where it got a much better response than I expected (300+ upvotes, great questions, zero roasting) GitHub is

Hey everyone! I’ve been working on a 100% local cognitive memory engine designed specifically to give multi-agent systems and NPC societies human-like memory. It's called Eternal Thread. Instead of ju

It worked pretty well, digging through and analyzing some local files. Claude code regularly stops at some point and fails to continue while DSH worked for 2 h, recognized that it could benefit from r

similar to video upscaler LLMs, what's the bleeding edge right now on something similar for audio music? the idea is you feed it in with a low bitrate MONO old soundtrack, and the LLM then recreates i

Sounds quite interesting, a big IDE provider optimizing for local AI with their coding harness. Especially that they picked Qwen3.6 over Qwen3.8 because of the thinking needs. Haven't read the full ar

Minimax just announced (only over xhitter apparently) that it is raising rates by 60%-65% tomorrow, the 25th. If you use Minimax, you'll either want to lock in their current rates. Sadly, every chines

MobileMoE is a family of on-device Mixture-of-Experts (MoE) language models with sub-billion active parameters, designed to push the quality–efficiency Pareto frontier for on-device LLMs, including th

An AQuA research loop let one LLM write an intraday trading feature and a second LLM review it for causality. The feature divided the volume seen so far by the day's final total volume. At prediction

ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning Large Language Models (LLMs) have demonstrated remarkable abilities in tackling a wide range of c

TL;DR: I'm planning to spend around 100 on cloud GPUs to benchmark Qwen3.8-27B with a focus on questions that actually matter when running it locally: different quant levels/providers, 8-bit vs 16-bit

If you’re trying to run local LLMs on a normal laptop, an older desktop, integrated graphics, limited VRAM, or simply the hardware you already own, r/LowEndLocalAI is meant for you. The idea is simple

For the GPU poor among us: I'm curious what results you're getting with low quants of Qwen 27B 3.8. My main inference hardware is limited (Mac mini M4 24 GB), but I'm getting great results with Unslot

I ran the Aider benchmark on Qwen 3.8 27B FP8 with FP8 KV cache 256K context vLLM. The score: 72.9 This matches Gemini 2.5 Pro from 2025-04-12 which also scored 72.9. Beats Claude Opus 4 from 2025-05-

I commented on another Qwen 3.8 27B post that I was frustrated getting anything to work. You all gave some great comments. I nuked openwebui and straightened out my llama.cpp docker config. 1 hour of

Ornith does really well. TielCoder (https://llm-bench.io/benchmarks/cmt7kp2zj002r01lcmpchvlko) might be even a bit better in coding. Will give it a try soon. Details of the comparison see here: https:

With vision enabled I am able to run at 150k context on a single RTX 3090 and the results are just amazing. I was even able to write a gmail plugin for DeepSeek harness with locally hosted Qwen 3.8 27

I'm one of the co-authors of this paper. It's a seminal work in exploring the properties of 3D generated by LLMs via spatial programming. I've set up visual demonstrations of such 3D objects at: https

Thanks to Unsloth Dynamic 3.0 quants coming in slightly leaner and better preserved, I settled on Qwen 3.8 27B (`UD_Q4_K_XL`) at 100K context as my daily driver for Hermes Agent and OpenCode. On an RT

It’s SO DAMN CLOSE. My harness has brought such a sense of calm whenever I sit and get some work done. Can't wait to get this out there. AMA, and yes, I've created a coding and cross domain harness an

Hi, Back in late march I begun playing around with Qwen 3.6 27b and found like everyone else that it's notoriously good at tool calls, where every model I tried before just derailed after a few turns

Try it here: https://ocean.blackbeardlabs.dev/ Model: Qwen 3.8 27b Q8_X_KL Unsloth Hardware: 3 x RTX3090 Harness: DeepSeek Harness Prompt: /goal I want you to create a **JavaScript + Node.js WebGL pro

Qwen3.8-27B is amazing, but it’s slow. A stronger 35B-A3B Mixture of Experts-coder that can run and solve real codebase issues fast (even on constrained hardware) is a valuable addition to the arsenal

The latest updates in Ollama have dropped support for the Pascal line of NVIDIA cards. I have a laptop with an NVIDIA 1070 which is no longer used by Ollama. Does anyone know of other frameworks like

I'm curious what other people are using for local LLM coding / agentic coding with only 8 GB of VRAM. My current setup is: Intel Core i7-11800H RTX 3070 Laptop, 8 GB VRAM 32 GB DDR4 RAM openSUSE Tumbl

Given OpenRouter.ai was snapped up by Stripe, who do we think would go after the 'GitHib' of AI models? It is a big chunk of change they are looking ($13B). Apple may be a contender to give them a rea

been building ShardFlow for the past few months, a distributed LLM inference framework that splits any HuggingFace transformer across N GPU machines and uses neural speculative decoding to deal with W

A few weeks ago I shared Agent Quest, my open-source experiment that turns Claude Code and Codex sessions into heroes living inside a small 2D world. The original idea was mainly about making it easie

WARNING: I think I made a mistake with omlx, and its results below seem to be without any kind of acceleration. I am rerunning them with Lightning MTP and ANE support for prefill. Will update the post

I haven't had a lot of luck in the reliability of the output of any models so far for local coding use, and find myself relying on Claude's limited free sessions to assist with coding. My coding appli

Hey r/Ollama, If you run local models on mid-range hardware, you probably know how fast VRAM disappears when you try to set up document memory or local RAG pipelines. I built Hillock (https://github.c

I haven't seen a post about this here, and I'm curious what you think about it. On August 14 Civitai rolled out the option for creators to choose 'permanent paid access - selling with no time cap'. Pr

This is something I've noticed. Back when GLM 5.2 and Kimi K3 launched, there was a media push on pushing how dangerous open source models are. We know why they were doing this; these open models are

I don't know about you guys, but Deep-seek harness is insane. It's not focused on being a coder agent, it's webUI made it very easy to just checkin from time to time, and the best part? Why it's bette

This is one more vibe slopped custom optimization for, in this case, my hardware (m2 ultra 60 cores, 192gb). It is just a fork from llama.cpp with a few changes, it achieves: - DeepSeek V4 Flash, no k

I've tried a few Gemma 4 models, for example 'gemma4:26b-mlx'. It says 'I cannot process images.'. On their website it mentions images, audio etc. https://ollama.com/library/gemma4/tags I’m using Olla

I'm running Qwen 3.8 27B locally on a single GPU. I can push the context to 131k, but I'd rather run it faster at 64k if the agent can manage context properly. What I have in mind is pretty simple: on