AI Wiki — Knowledge Base

I turned Qwen3.8-27B Q2_64 + llama.cpp into a fully TypeSafe AI-compatible Jev-like system. OpenAI API still intact! World’s first Vision-enabled Jev-like model! <10 GB VRAM, 170 ms on an RTX 3090 and ~140 tok/s in chat. 76% vs. 88% Jev-1.13 Acc. on a diverse 22,000-request typed-decision benchmark

Yesterday I released Bonsai-Llama-Jev - a Jev-like Typed Decision inference system for locally operated, sovereign AI. In my new typed-decision-bench, covering more than 22,000 decisions across 175 us

model-releasesr-localllama

23 Sept 2026

Search: “inference”

Question: Why is prefill unbelievably faster in vLLM than other inference engines?

New: Llama.cpp adaptive speculation for faster inference

SenseNova U1.5-Lite full release: expert training, OPD distillation, one model at inference

CPU-only inference on a Celeron N5095 SBC: 6 models from 0.6B to 8B, benchmarked

Using Absorbing Markov Chains + EmbeddingGemma 2 as a Decision Engine: ~2ms inference with 100% GTO Parity on Vegas Blackjack

Epyc inference rigs

Looking for developer-friendly inference providers who give you enough API credits to experiment [D]

poorman inference engine for 16GB GPU and 35B moe Qwen 3.6for coding

If you are running Qwen 3.8 Flash Next on Strix Halo, use this software for inference. It's so much faster than llama.cpp especially at high context.

Hardware Roofline Inference Calculator

Finally got qwen 3.8 q4 k_M 27b runnin slick 40tkps load at 240k inf 4qnl for stable agentic coding on a 7900xtx. I have been finally able to give hermes a task and come back to results. Built a panel to manage my inference servers from hermes.

Call for compute - help optimize Qwen inference speed on local hardware

Swarm inference AI

Thinking about grabbing an RTX 2000 Ada 16gb to add to my gaming pc for inference due to Wattage constraints, any advice?

Perplexity open-sourced their Mac inference server for Qwen 3.6

Visionary — Krea 2, MiniMax-H3, inference and LoRA training in one app

Has anyone here fiddled with TPUs for inference ?

AMD Acquires Taalas to Advance Compute Solutions for Rapidly Growing AI Inference Market

Vendor-agnostic ML inference on production edge devices [R]

Qwen3.8-27B: Using KV Cache Transplants to Boost Output Quality

Tired of bloated UIs for Ollama? Built a minimal IDE that infers your hardware and just works

Faster than Light in Air: 8-22 tg/s Qwen3.8-Flash-Next (Q4/Q4ish) on a 32GB M4 MacBook Air

Flama: 8 years building a Python framework where ML models and LLMs are first-class, now written up on arXiv [P]

Qwen3.8 flash next ISTA-DASLab GGUF 50t/s TG and 1500t/s PP with 12GB VRAM and 64GB RAM Laptop on 'Strata' engine

The CEA architecture is a bigger deal than I initially thought

Continued development of the model based on the SSN [D]

Building a budget 32GB → 48GB VRAM home AI server: 2-3x RX 9060 XT 16GB vs RTX 5060 Ti 16GB, AM5 vs used EPYC?

What’s the community’s favorite benchmark to validate performance?

What are people using for low-latency autocomplete in production? [P]

openbmb/MiniCPM-V-4.7-35B-A3B

[Benchmark] Running Local LLMs on Orange Pi 5 Plus (RK3588, 16GB): Ollama Tok/s, NPU Offloading, Core Pinning & Thermals

CONFIRMED: Ollama Cloud nerfing usage limits on grandfathered Pro/Max Plans

Got MiniMax H3 video generation running in TensorSharp

I pushed Qwen3.8-27B to 124 tps on a single request on a RTX 3090

300b on 32gb MoE-streaming findings + optimisations

Has anyone been working on a solid setup for DSV4F on x2+ R9700s?

Open Source Ternary LLM Engine in Rust/CUDA for Quantization, Serving, and Training of models on consumer GPUs, called Tritium (Apache 2.0)

Benchmarks: TensorSharp vs. llama.cpp

RTX 5090 users: TensorRT-LLM vs llama.cpp (GGUF) for Coding Agents (Cline/RooCode) – Is the speed worth the VRAM limit?

openbmb/MiniCPM-V-4.7-1B

Final revision of zzpanic vllm-radiance for 1 x R9700 GPU, qwen3.8-27b

Deepseek Peak/Off-Peak and Ownership

Qwen3.5 arch implementation in FPGA fabric for 9B/27B INT4 models on relatively cheap eBay mining hardware

Two 96 GB Ascend cards crun Qwen3.8-flash-next hardware notes, vLLM work, benchmarks, and what is next

50B+ MoEs with few active parameters, what's the sweet spot for intelligence, agent speed, and affordable fine-tuning?

Don't trust frontier models when asking about budget hardware!

VLLM-MLX vs Ollama MLX... Ran concurrency benching. Qwen3.5-4B.