Search: “routing”

This Reddit post discusses a known instability in PPO when advantage estimates operating across different temporal scales (e.g., short-horizon and long-horizon returns) are dynamically routed or mixed

Training large language models (LLMs) is resource-intensive, and adapting them for diverse deployment scenarios with varying computational constraints remains challenging. While elastic architectures

Hey r/ollama 👋 If you run Ollama locally alongside cloud endpoints for agent workflows, Cursor/Windsurf, or custom scripts, managing API switching, failover logic, and context limits can get messy fas

I have been building a somewhat unusual local inference machine around two Huawei Atlas 300I Duo cards. They are relatively inexpensive, passive, dual-accelerator PCIe cards with 96 GB of device memor

I’ve been experimenting with a custom inference/runtime research project called CRANE V2, mostly because I wanted to answer a stupid question: How far can you push absurdly large MoE models on an ordi

ik ik 'another harness D:'. so... i spent the last 6 months of my life building and ironing out every single detail and bug through almost every route with extensive testing across 3 machines, daily,

Sentinel is a local-first AI coding desktop application built with Rust and Tauri that automatically routes coding tasks to appropriate models based on task complexity. Instead of swapping between mul

This is an update. I posted LocalMind here many moons ago from another account, when it was a Gemma chat in a tab. LocalMind is a static web page that runs models on your GPU through WebGPU. It has no

We built an inference engine for MoE models that don't fit in VRAM + RAM. Most of the model stays on the SSD, and experts are read as tokens need them. This started as a proof of concept, and poc work

I run a personal always-on agent fleet — eight seats claiming work in parallel from a shared task board, routing across Anthropic, OpenAI, xAI, Google, and a local Ollama floor. This week the spend fo

I've been running Qwen3.8-Flash-Next as my main local coding model from past few weeks. The file is 95.5 GiB and my Mac has 64 GB. It works because the routed experts stay on SSD and only get read whe

ChatGPT is usually treated as a prompt box. Ask for a cinematic shot, add a few camera words, copy the result into a video model, and hope the ending survives. My problem was that the ending usually d

I wanted to find out whether a huge text-only MoE could be given basic vision without retraining the language model itself. The short answer is yes. I froze DeepSeek V4 Flash and a 417M-parameter Moon

TensorSharp's MoE CPU-offload feature has been merged into main. Here is the parameters description of this feature: Mixture-of-Experts CPU offload: --n-cpu-moe <N> | -ncmoe <N> Keep the routed MoE ex

A few people asked for this after the bandwidth thread, so here it is on its own instead of buried in a comment. The dense rule was simple: every token reads every weight, so tokens/sec ≈ bandwidth ÷

Everyone is talking about Kimi K3, but if you jump straight into the technical report, you’ll quickly realize it’s standing on years of research -- just like any breakthrough is! If you want to unders

Hey HN, Henry & Roman here from Cactus. A small, on-device model is fast and private, but sometimes wrong, but frontier models are getting expensive pretty fast. So, we post-trained Gemma 4 E2B post-t

ALHR - Adaptive Learnable Hierarchical Routing, uses static binary trees and learnable functions to minimize the amount of keys to be read. It does use a dense teacher while phase 1 of training howeve

I spent the first week of September building a personal assistant over Gemma 4 in Ollama on a 16 GB M1 Pro: Node backend, React web UI, Expo phone client, two-model routing, cross-session memory, RAG

Kimi routed some PLA requests to Claude for distillation purposes without warning the PLA users. There is rumor that 16 Moonshot employees were arrested for this leak. submitted by /u/Ok_Warning2146 [

On September 1st, three young men from California used Gemini to plan a hiking route on Mount Shasta. The AI ​​told them to bring far less water and food than was actually needed for a hike of that di

Quick update on the RTX 5060 Ti local LLM repo. It has changed quite a bit since my previous posts. The project started as a collection of practical notes and benchmark results. That was useful, but a

I managed to get Qwen3.8-2.4T-A95B running locally with llama.cpp on mu PC just for fun, cause why not. I was using the Unsloth Qwen3.8-2.4T-A95B-UD-Q1_0 GGUF quantization. The full GGUF is about 397

Inspired by a post from u/giveen I motivated claude (no patinence on my side to work through everything myself) to help me get DS running on my MacBook M5 Pro 64GB and it exceeded my expectations.. be

I serve a free local model on my Mac Mini and route real agent work to it. To check I was not fooling myself, I set up a blind grader that replays frontier tasks locally and scores both. https://previ

I've been building Krasis, an MoE-focused runtime for streaming big models through limited VRAM on NVIDIA consumer/workstation GPUs, and I think this is the most interesting result so far: Ornith-1.0-

Qwen3.6-27B: SFT vs continued pre-training vs RL? I’m interested in adapting Qwen3.6-27B, but I’m increasingly unsure whether conventional SFT/LoRA is the best route if the goal is to add a capability

We spent the last months consolidating seven separate sequence classifiers into one multi-head model, our apex model, so to speak, and since the weights are now public, I wanted to share what worked a

Manifest is a tool or platform that has added support for Ollama Cloud, enabling users to route AI model inference to cloud-hosted Ollama instances rather than relying solely on local GPU resources. T

A developer shared on r/ollama how they built a custom VS Code extension that routes Claude API requests through locally-run models via Ollama, dramatically reducing cloud API costs to approximately $

A Reddit post from r/ollama describing how a user dramatically reduced their Claude Code token consumption by 178x, likely by routing simpler or lower-stakes tasks to a locally-run model via Ollama in

Disclosure first: I'm one of the maintainers, this is my project. If you've pointed Open WebUI or a script at a local server, you know the failure modes: the server restarts and every conversation is

I've been tinkering with GLM-5.3-Flash on two 64GB CMP 170HX cards for a while, and the setup is finally stable enough that I figured I'd share it. I also compared it against the Qwen3.8-Flash-Next se

I forked llama.cpp's server into AgrillaMoE, a dedicated build for Qwen3.6-35B-A3B (~A4B) with Unsloth quants. On a (vant.ai) rented V100 16GB with the 2-bit UD-Q2_K_XL quant it generates at ~57-60 to

Our agent workflow is currently executing a group of 10 vllm api calls within a asyncio.gather we made them share the same prompt until the end where the queries/instruction prompts differ. These call

We have released Qwen3.8-Flash-Next quantized with GSQ and RCO, together with a second, capability-targeted build in which half of the model's experts have been removed. Flash-Next is a sparse mixture

I’ve been experimenting with whether Qwen3.8-Flash-Next’s pretrained PLE n-gram memory can improve a much smaller Qwen3.5-0.8B model. I trained the 0.8B setup with limited resources, mostly using free

Hi all! After all the jev hype I wanted to try by myself and got qwen3.8 27b to mix-and-match open implementations on top of ninfer, and to be honest I'm quite impressed with the results, latency is n

Hi Ollama Team, I’m writing regarding the announced retirement of GLM-5.1 from Ollama Cloud on 25 September 2026. I wanted to provide some fairly detailed technical feedback because GLM-5.1 is not sim

GitHub: https://github.com/pfekin/LARA I've been working on LARA (Lightweight Additive Residual Adaptation), a research project on making post-training modular for frozen language models. I've also de

Hey everyone — I’m building TensorSharp, an open-source LLM inference engine. Here are the latest DeepSeek V4.1 Flash GGUF results using its native ggml_cuda backend. Setup: 8× NVIDIA A40, layer split

I wanted to see how my fully local home voice assistant compared to the latest GPT Live, so I tested it using the same conversation used in their 'Improved Intelligence' demo. In this video they ask t

Original Source from DeepSeek WeChat Official Account: https://mp.weixin.qq.com/s/qg0NU3NNUbp1co2PdkAPAg Today we're officially releasing the DeepSeek V4.1 Flash model. It is the smallest model in our

Megathread for discussing the release of GLM-5.3-Flash. Quants Fine-Tunes & Abliterations Chat Templates Inference Server Support & Configuration Experiences, Benchmarks & Model Comparisons We'll try

My best bet is the embedding-offloaded linear where the 51b n-grams track semantics and context, like 3.8 with its loss of real world knowledge bolted back on. Otherwise? Sparse full-attention with de

MobileMoE is a family of on-device Mixture-of-Experts (MoE) language models with sub-billion active parameters, designed to push the quality–efficiency Pareto frontier for on-device LLMs, including th

I've been building AgentGraph Studio, a visual editor for designing multi-agent workflows and exporting them as runnable Python code. One thing I wanted to verify was whether the generated code actual