Search: “inference”

I’m running Ollama on an M4 Pro Mac with 48 GB unified memory and testing local coding models for repository work. Direct Ollama inference is fine. Qwen3.8 27B-MLX runs well enough for me, and the loc

Hi, After a long night of optimizations, I believe I have made the fastest inference engine for Qwen3.6-28B on a 3090. Quick metrics: - 250w power capped - Up to 195k context (ships with 150k for safe

My local inference build, with: ASUS WS C422 PRO/SE 10-core Xeon W-2255 64GB ECC RAM 64GB VRAM Pimped case with TurboLEDz indicating the frequencies of the 10 xeon cores. Running llama.cpp with SYCL b

Megathread to help with the influx of duplicate / similar posts around the release of the Qwen 3.8 27B release. Quants Fine-Tunes & Abliterations Chat Templates Inference Server Support & Configuratio

LFM2.5-VL-3B is a multimodal variant of LFM2.5, a family of hybrid models designed for on-device deployment. It builds on LFM2-VL-3B with further mid- and post-training. LFM2.5-VL-3B can process both

I wanted to find out whether a huge text-only MoE could be given basic vision without retraining the language model itself. The short answer is yes. I froze DeepSeek V4 Flash and a 417M-parameter Moon

VLX-Seek-1.5-10B VLX-Seek-1.5-10B is the open-source 10B model in the VLX-Seek 1.5 family, designed for fine-grained perception and visual grounding in embodied scenarios. It targets practical setting

Demand for low-precision inference, including NVFP4-based approaches, has grown as large language models are increasingly deployed in latency and cost constrained production environments. Quantization

Hey everyone, I constantly see high praise for M3 and M4 Macs for local LLM inference, even the base/16GB models. However, my experience has been quite different, and I'm trying to figure out if I hav

seems to be about as good as a vega 56 with 16Gb of VRAM, is it worth it? (don’t want to deal with NVIDIA drivers on Linux, already have an rx6650xt and might simply use vulkan for llamacpp inference)

High-performance inference of NVIDIA's Parakeet TDT 0.6B V2 English transcription model, in the browser. Check out the live demo: https://parakeet.narcotic.sh/ A fully custom, dependancy-free implemen

As you all know the model is 2.69B parameters with a 128K context window and purpose-built for multi-step agent workflows. What you are seeing is the Q4_K_M GGUF running on my own inference engine bui

I've been out of these forums for awhile but I figured I would provide a formal update on how this has been going now that it has some operation time under its belt, just to put the information out th

I deployed K3 on 32 H100s at work a couple of weeks ago and then got annoyed that there was no way to poke at it on my own machine. So I wrote an inference engine for it in C99. Nothing clever going o

So I am in a conundrum and I'm thinking of asking for your opinion for the following: Currently, I have a 5800X3D gaming rig with a 7900XTX with its 24GB VRAM. It seems that for this subreddit, this c

I'm an avid user of Deepseek v4 Flash via antirez's DS4 DwarfStar inference engine, and so when the new checkpoint dropped, the first thing I did was rent a cloud box and spin up a quantization for us

Hi all, I have not seen this model talked about here but it seems like baseten (inference provider on OpenRouter) merged the vision encoder from Kimi k2.6 into GLM 5.2. I think the lack of vision was

If you were able to get an inference machine that could do decode at 1k toks/s or even 10k tok/s, would that even be helpful? Would it unlock any new use cases? Let’s assume that this is for actually

I've been digging into Krea2 to see if there's any way to condition inference in specific regions of pixel -> latent space to implement bbox type prompting in order to apply multiple simultaneous char

What quantization or storage size tiers should we expect further down the line? This site suggests q4 and q8 are coming. But I don't know how to translate that to storage size for local inference. htt

Hey r/LocalLLaMA — solo researcher here. I built a new attention architecture and want independent testers. Wave Field LLM replaces O(N²) dot-product attention with FFT wave convolution on a field. Tr

The Mac mini M4 uses unified memory architecture where CPU and GPU share a single 24GB memory pool, while the RTX 5060 has dedicated VRAM. Mac mini M4 is preferred for large model inference (70B param

A user reports successfully running Qwen 3.6 35b MoE (mixture of experts) with Zoo Code on an M1 Max Mac, achieving local inference without external servers. The setup enables fully local, battery-pow

This post discusses how OpenAI is burning 17 billion annually despite 20+ billion in revenue because unit economics for AI inference are fundamentally broken, with the company losing more money on que

OpenStudio is a hybrid AI router that combines local model inference with cloud-based model access through OpenRouter, a unified API providing access to hundreds of AI models through a single endpoint

Setting a GPU power limit using nvidia-smi reduces heat output by approximately 20% with only a 5-8% inference speed loss. Undervolting the GPU can reduce power consumption by 5-15% with zero performa

This Reddit post likely compares inference performance metrics across popular language models running on Ollama, measuring tokens per second as a key performance indicator. The post would help users a

ZIT (Z-Image Turbo) is a compact, fast distilled model tuned for photorealism that uses 8 inference steps with a fixed CFG of 1, which limits creativity . ZIB (Z-Image Base) is an undistilled model id

Ollama Cloud launched in September 2025 with fixed-price subscription tiers (20/month Pro, 100/month Max) for cloud-hosted inference , while local deployment remains free with unlimited local usage .

An MCP bridge project that enables integration between AI coding tools (Kiro, Claude, and Cursor) and local Ollama instances for offline model inference. The bridge facilitates communication between t

A Reddit post from r/StableDiffusion in which a community member shares their experience building a custom real-time telemetry dashboard to monitor inference metrics for the LTX 2.3 AI video generatio

This Reddit thread likely discusses user frustrations with performance issues when running GLM-5.1 via Ollama's cloud inference option (`glm-5.1:cloud`), a flagship agentic coding model from Z.AI. Com

IC-LoRA (In-Context LoRA) enables conditioning video generation on reference video frames at inference time, allowing fine-grained video-to-video control on top of a text-to-video base model. Unlike t

Manifest is a tool or platform that has added support for Ollama Cloud, enabling users to route AI model inference to cloud-hosted Ollama instances rather than relying solely on local GPU resources. T

This Reddit post on r/MachineLearning presents a community-contributed dictionary of contemporary Large Language Model (LLM) terminology, covering terms related to training, fine-tuning, inference, al

This r/StableDiffusion post discusses community experimentation with custom sigma schedules for the LTX-2.3 Distilled video generation model — a fast-inference checkpoint that completes video generati

Users in the r/ollama community reported that Ollama reduced the usage limits on their Pro subscription tier for cloud-hosted inference. Ollama Cloud is available through subscription tiers including

TurboOCR is a high-performance OCR project that combines PaddleOCR with NVIDIA TensorRT, implemented in C++ and CUDA, achieving throughput of 270–1,200 images per second using FP16 half-precision infe

A Reddit post on r/StableDiffusion showcasing image generations using a combination of Z-Image Turbo — a distilled 6B parameter efficient image generation model with sub-second inference latency — pai

A developer shared an open-source Android keyboard on r/ollama that integrates local AI capabilities by connecting to self-hosted inference servers such as Ollama, LM Studio, or any OpenAI-compatible

Ollama Cloud offers Free, Pro ($20/month), and Max ($100/month) subscription tiers for cloud-hosted inference, but token generation speed depends on model size, architecture, and hardware optimiza...

The ASUS ROG Flow Z13 (2025) with AMD Ryzen AI Max+ 395 (Strix Halo) and 128GB of unified LPDDR5X memory is a capable portable option for running large LLMs locally, with ASUS officially stating it...

The Mac Mini M4 Pro with 48GB unified memory is a popular choice in the local AI community for running large language models via Ollama, as its Apple Silicon architecture makes all 48GB of RAM dire...

Local AI tools such as Ollama, LM Studio, and Jan all rely on the same underlying inference engine (llama.cpp) and support compatible model formats (primarily GGUF), meaning a single downloaded mod...

I saw the 4x R9700 build for a 10-person startup here a few days ago and wanted to share my results, since I opted to spend most of the money on the cards. I went with a super cheap used EPYC instead

Hey r/LocalLlama - Today we’re excited to release Muse Spark Open, the most capable US open-weight model to date. Built for coding and agentic work, it excels at long-horizon tasks, complex software e