Search: “deployment”

audio.cpp 0.7 is out :) This release adds a lot of new audio models and a new way to compare them locally. Audio.cpp is now at 62 model families and 85+ model variants. And it keeps growing! The bigge

TL;DR - A ~4M example, ~3B token CoT dataset designed around helping small models think more concisely, accurately and reliably. Hi all! For the past few months I have been working on a dataset design

after 3 months and 800 burned... Unbounded Labs is proud to introduce Bart, our vintage LLM: 2.82B parameters trained from scratch on 20.1B tokens of English written before 1931. You can talk to it ri

ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning Large Language Models (LLMs) have demonstrated remarkable abilities in tackling a wide range of c

What I ran: 8x B300 on Modal, 56.79 per hour, vLLM, tensor parallel 8, native MXFP4 Cold boot ~27 min (1.56 TB load, JIT, 51 CUDA graph captures) TTFT 0.92 to 1.02 s, decode 92 tok/s steady, 83 tok/s

Hi everyone, I'm currently working on optimizing an LLM setup at my company and I'm trying to identify where the actual performance bottleneck is. Our current setup is roughly: LLM: NVIDIA Nemotron mo

I have been searching for suitable model to run on my 8GB RAM toy, NVIDIA Orin Nano Super 8GB. This little toy was priced at 249 earlier this year (not any more), and pulls very little power when idle

There’s a hands-on workshop on August 29 that builds and benchmarks this properly, end to end, using entirely open models, no API calls involved. Led by Ben Auffarth, AI Consultant and Founder of Chel

North Micro Vision Instruct is a 2.4B-parameter open-weight vision-language model with native-resolution image support, released under the Apache 2.0 license. It is designed as a compact foundation fo

Hey guys, Just finished benchmarking DeepSeek V4 Flash 284B + DSpark on a single RTX PRO 6000 96GB. Short version: DSpark: ~15–17% faster generation on my coding workload On this setup, the DSpark dra

Hi LocalLlama, we're super excited to release Unsloth Desktop today! 🦥 It's the first desktop app that enables you to run and train models locally. Open-source. Available on Mac, Windows, and Linux Su

There's an interactive chart and some extra data in the blog post if you're interested. There are plenty of KL-divergence benchmarks for GGUF models, but most of them compare one GGUF quant against an

Spec-dec has been a thing for a while, in fact, it's wasn't an idea that was born for LLM inference. E.g. Uber's https://github.com/uber/submitqueue applied it to a merge queue. Apple & GDM had been r

I'm the author, so discount the enthusiasm accordingly. This is an unaffiliated community port, not endorsed by the vLLM project, which it uses to verify its correctness. What started it: I love vLLM,

Questions & Responses(in BOLD) below. Favorite question(s) moved to end of the thread with combined responses(removed duplicates). Be optimistic folks. I'm sure we're getting other models too apart fr

The video is one real run, not a mock-up: grapharc go 'why did checkout latency spike at 09:14 UTC?' --model ollama/qwen3:8b A local 8B model proposes the graph → triage fanning out into four parallel

I've been out of these forums for awhile but I figured I would provide a formal update on how this has been going now that it has some operation time under its belt, just to put the information out th

I have been working on this tool for months and there are a lot of new functionalities and tests that are going to be released in the next few weeks! The goal of the tool is to allow community members

Hey everyone, I’ve built a Windows Middleware Agent Server designed to pair local LLMs (via Ollama) with frontends like Reins App. Key Features: Self-Healing Loop: If a generated PowerShell command fa

I'm an avid user of Deepseek v4 Flash via antirez's DS4 DwarfStar inference engine, and so when the new checkpoint dropped, the first thing I did was rent a cloud box and spin up a quantization for us

Hey folks, is anyone currently using Ornith 1.0 9B for serious or complex backend development? I’m working on three different applications: A multi-vendor e-commerce platform with a Node.js backend A

This post was not written by a clanker. Hey guys, I'm a comp sci major who wanted to introduce a cool project I built for quantizing models to ternary (1.58 bit) with as minimal of loss as possible, a

Hi everyone! We’ve just released a major update to the leaderboard! We are expanding beyond Python with a new multilingual slice featuring real-world software engineering tasks across 5 languages. Ope

Hey r/LocalLLaMA, I’m a retired engineer with a background in distributed computing, currently running a 1-person startup. Like many people here, I’d love to experiment with 2T+ MoE models locally. Th

I was refreshing my youtube and found out my favourite reviewer uploaded a battery test of 78 smartphones: https://youtu.be/MpgUFrsIWSQ the author said they started using robotic arm to simulate a per

tests done power limiting the GPU on LACT - real power usage varies wildy at 20W it ranges from 25W to 56W same behavior happens on every setting prompt for the test runs: https://github.com/lukesdevl

Built an open-source AI coding agent that was 7%–75% cheaper than a cold 'claude -p' run on 6/6 well-localized tasks across repositories up to ~82k LOC. The biggest difference: Cold agent: 6.83, 207 t

In the past few months I’ve experimenting heavily and tortured my old 2016 Desktop PC to run the biggest Local LLM I can fit. I documented the whole process and research and I’ve published a repositor

One thing I noticed in American politics, whenever the government wants to push unpopular actions or laws, they often introduce fear to convince the public to support them. This is actually how i view

What it is: ATH-MaaS/OvisOCR2 - a 0.8B document-parsing VLM post-trained from Qwen3.5-0.8B (SFT + RL + OPD), Apache 2.0, runs on vLLM 0.22.1. One prompt per page image -> complete markdown (HTML table

This handbook covers the technical aspects of running large language models efficiently at scale, focusing on GPU optimization techniques including GPU internals, key-value (KV) cache management, batc

Forge Neo is an upgraded version of Forge UI built on the Gradio library , a lightweight Stable Diffusion web interface for image and video generation. It supports multiple new diffusion models includ

A developer documented their experience building a fully-local paper Retrieval-Augmented Generation (RAG) system using two NVIDIA GTX 1080 Ti GPUs and one RTX 3090, sharing three significant challenge

MiMo-V2.5-Pro is a model available on Hugging Face that was requested to be added to Ollama's cloud models in May 2026. The discussion on r/ollama likely covers the availability status of this Xiaomi-

An updated MarkItDown API Server integrates Microsoft MarkItDown for converting PDF files, images, and Word documents to Markdown, with Ollama and LLaVA for generating image descriptions. This project

This call for papers announces a workshop focused on the growing need for efficient and effective techniques for editing trained models, especially large generative models. The workshop solicits paper

This post describes implementing DCGAN (Deep Convolutional Generative Adversarial Network) inference on resource-constrained microcontroller hardware, achieving image generation with a 12.6 million pa

Ollama has added MLX support for Qwen3.5 and Gemma 4 , with both MLX and NVFP4 quantization variants available for these models. There have been issues with broken Qwen3.6 NVFP4 generation on Ollama ,

NuExtract3 is a unified 4B vision-language reasoning model for document understanding that combines structured information extraction with image-to-Markdown conversion, suitable for OCR and RAG prepro

A discussion from the r/ollama community about using local large language models for privacy-sensitive applications, particularly for handling sensitive documents like medical records. Local LLMs proc

G4-MeroMero-31B-uncensored-heretic is a fine-tuned variant of Gemma 4 31B optimized for creative tasks, featuring low KL divergence (0.0100) and minimal refusals (15/100). The model is designed to be

This discussion explores which language models work best when combined with RAG (Retrieval-Augmented Generation) for building a locally hosted LLM focused on survival and off-grid living topics. The t

This discussion explores using Ollama to run large language models locally for invoice processing while maintaining control over data. The thread likely addresses selecting appropriate smaller models

This discussion thread explores lightweight vision-language models that can run locally, including options like Llama 3.2 Vision, Qwen2.5-VL, and SmolVLM2, optimized for tasks like OCR and visual ques

This post benchmarks three Qwen3.6 model variants against gpt-oss:120b when running on Apple Silicon hardware, evaluating their performance characteristics and practical usability. It documents both t

A community repository of AI agent configurations for Ollama has reached 888 GitHub stars, featuring pre-built setups and configurations for running AI agents locally. The project appears to focus on