AI Wiki — Knowledge Base

CrofAI 'cheapest inference provider in the world' gets exposed as an OpenRouter wrapper, routing requests to smaller, cheaper models at up to 20x markup. CrofAI responds to Wire Fraud allegations by denying everything, then backtracking, then 3 hours later wiping their entire online presence

Disclaimer: no AI was used whatsoever to write this post Cautionary tale about chasing cheap tokens. exposé: https://kendell.dev/blog/crofaifalse/ reaction by nahcrof, announcing the shutdown of the s

model-releasesr-localllama

15 Sept 2026

Search: “inference”

SOTA Apple Silicon Inference (August 15, 2026)

Inference Engines will become a series of one-offs

Inference Engineering for Dummies

I implemented the YOLO26n model inference from scratch using ARM64 Assembly Language (No framework) [P]

GitHub - google-ai-edge/ml-drift: GPU-Accelerated AI/ML Inference

Running an LLM-driven town with 800+ persistent agents: concurrency, context caching, and inference costs

p2p ai inference mesh

The Rise of Overfit Inference Engines

llama.cpp Open PRs list - CPU/RAM/Disk/Hybrid Related - Better for CPU-only & Hybrid inference

Semi Edge Inference Idea [D]

Understanding GPU Inference Workloads [D]

DKV: Open-source KV-cache compression framework for local LLM inference (CLI + technical report)

Java vllm-like framework claims 90% of perfomance of llama.cpp on local inference on NVIDIA GPUs by compiling Java to CUDA and cuTile

Forlinx 20-TOPS M.2 AI accelerator supports PCIe cascading for local LLM inference

2400cc Inference Racer: Dual RTX 3090 motors, NVLink turbo, naked 7840U ThinkPad ECU, VW Golf radiator

CrofAI 'cheapest inference provider in the world' gets exposed as an OpenRouter wrapper, routing requests to smaller, cheaper models at up to 20x markup. CrofAI responds to Wire Fraud allegations by denying everything, then backtracking, then 3 hours later wiping their entire online presence

Are inference providers able to make any margins?

VoxGen, an AMD-optimized TTS inference engine for VoxCPM 2 models

I built a DwarfStar-inspired Vulkan/Metal inference engine for Qwen3.6-35B-A3B on 16 GB machines

LLM Inference Dashboard

jinfer: An open-source AI inference engine for the JVM. Finally, AI in jar.

What are your best sources to learn about LLM inference?

Building a zero-dependency C inference engine for BitNet (1.58-bit) - lessons from hitting 36 tok/s on a Xeon CPU

DCGAN inference on a microcontroller: 12.6M parameters, 512KB SRAM, 26-second generation, pure C [P]

Best value upgrade path from 12GB VRAM RTX4080, 16GB system RAM gaming laptop for local LLM inference?

Stability Matrix Inference & seed usage

ASUS UGen300 USB AI Accelerator 8GB for local inference

Basalt: Fast Blackwell inference engine for Qwen 3.8 Flash Next (and how we got vision for free)

A Little Guide to Learning Distributed Algorithms for LLMS Training and Inference [D]

I added Qwen-Image 2.1 + LoRA support to TensorSharp (GGUF, local inference)

Does high / long term inference damages GPUs?

GLM-5.2 local inference: ubatch size made a much bigger difference than I expected

Does PCIe matter much for inference?

I ported vLLM's serving stack to C++20: 66 MiB binary, no Python at inference, output checked token-for-token against vLLM

Decrease the power limit of your 5090 to at least 480W - the performance penalty for inference is negligible.

How well do multiple GPUs scale for LLM inference? (Trying to understand the basics)

[Help] Gemma 4 26B LoRA Training on 16GB VRAM: Loss decreases, but inference degenerates into loops (Masking vs. MoE?)

[Paper] Stepped MoE: Segment-Level Routing with Configurable Inference Complexity

Trying to optimize LLM inference latency — Ollama + OpenWebUI/AnythingLLM

An open handbook on LLM inference at scale (GPU internals, KV cache, batching, vLLM/SGLang/TensorRT-LLM) [P]

Quantization and Fast Inference (MEAP) - How much performance are you actually getting from quantization in production? [D]

graphsignal-top: a `top` for inference GPUs in the open-source Graphsignal profiler, kernels ranked by time, live, with readable names

I built ALHR: A tree based sparse attention system that achieves sub-quadratic inference while retaining accuracy. [P]

Best current R9700 inference engine?

Qwen 3.8 flash next is based on Qwen 4 architecture, if the announced Qwen 4 27b is also the same architecture with n-grams does it mean I can actually have faster inference on a single 3090 without tweaking much?

Gewell - Gemma4 inference engine

Solved: LLM inference on Windows was 2–3x slower when the server window wasn't focused

We open-sourced Paddock, our Rust/C++ inference engine with its own CUDA kernels (MIT/Apache-2.0)