All entries

I have a tool that uses Gemini Flash with the lowest thinking budget to do summarization work. It's very fast, 0.9-1.2s in most cases. But I have a user experience problem where people make the wrong

We’ve released the complete VeriLoop E2 GGUF precision ladder, from the canonical BF16 reference down to a mixed-precision IQ1_M deployment tier. This post is specifically about the GGUF release and l

Hello! I am currently building my local AI server, I have the Huananzhi H12D-8D EPYC Motherboard with 8x16GB memory sticks at 2666 mhz (waiting for the other components at the moment) I currently have

I run a personal always-on agent fleet — eight seats claiming work in parallel from a shared task board, routing across Anthropic, OpenAI, xAI, Google, and a local Ollama floor. This week the spend fo

Hi people, I am looking for a lightweight IDE or plugin that won't inject large context at initiation. I tried Cline and native VS Code but they inject such heavy initial context that it fills up my g

​ Hi. I’m developing my first Python project (been at it a couple weeks). I built a very basic CLI ollama client. When a user sends a prompt I append the message to session['history'], save the sessio

I only just noticed this today and I’m genuinely surprised how useful it is. As of the September 23 update, ChatGPT Voice can now use the plugins and connected apps available to your account while you

Just watched a video from three years ago where Matt Berman tested Bard (yes, remember Bard?) and most of the tests were like summarization or logic tests. If it ever came to coding, it was the snake

Ok, so the Mac M5 Ultra (256GB) hit the market, but the only publicly available benchmarks material are flashy YouTube 'clown influencers' videos. We need serious numbers to evaluate whether Apple’s s

from FreedomIntelligence: HuatuoGPT-3-27B is a medical LLM built on Qwen3.8-27B with One-stage Policy Optimization (OnePO). OnePO adapts language models to medicine in a single reinforcement-learning

Hi - I want to build a home server for Ollama. I want to connect to the server via my laptop and mobile devices to do research and run agents. What is recommended to get this going ? submitted by /u/R

Built this because I got tired of guessing whether a model would fit in my 6GB VRAM before downloading it, and because ollama.com's library doesn't let you sort by size. What it does: - Benchmarks Oll

Hey r/Ollama, I've been working on Presentation Forge, an open-source presentation/report generator that supports Ollama for local inference. I originally made it because I'm a university student and

Hey everyone, A major issue when building test environments or fine-tuning pipelines is balancing data realism with privacy. Pasting real internal/client data into cloud endpoints is a massive complia

I’m releasing OpenJudgement-4B-Preview, an experimental Qwen-based model fine-tuned on custom datasets for classification, scoring and true/false judgments. It scores answer options directly, and Pyth

I was having trouble searching and analyzing the different models that Ollama has available ( https://ollama.com/library ). I wanted to filter for specific model sizes and tooling capabilities, so I b

Hi, sorry if this doesn't belong here. Getting to the point basically, I've been using online-only AI like GPT/Gemini since 2022, and have been interested in local models but am clueless overall. Yes

Just wanted to share with someone - don't have much to report yet. I am interested in this new AliceAI model and have been wanting to make a community impact for a while - and releasing an initial age

Original post: https://www.reddit.com/r/LocalLLaMA/comments/1woscea/contrastive_language_models/ (sorry I felt it wasn't giving CLM the highlight it deserves) What it is: a new projection head for Qwe

I’ve been experimenting with local AI lately, and one thing I’m finding interesting is what comes after getting a model running. Ollama makes local model access pretty straightforward, but for an actu

I'm about to buy a Mac Studio mainly for running LLMs locally and I'm stuck between two configs: M5 Ultra (30/64) with 96GB: 1.2 TB/s bandwidth, roughly 1.7x faster generation and much faster prefill

Hello Everyone! Good news for anyone who can compile code on apple, you can now experience using a AMD Radeon AI Pro R9700 on a MacBook Pro or Mac mini, Mac Studio or any OSX variant as long as it has

Model Overview Ling-3.0-flash-VL inherits the language, reasoning, and long-context capabilities of Ling-3.0-flash, while extending them with native image and video understanding. The model has 124B t

These boards cost me 115 each and I have them connected using llama.cpp with Vulkan and RPC on Bazzite. The boards have roughly 27GB of combined GPU memory and communicate over 1gb Ethernet. For aroun

Guys I have a M4-mac air with 24 GB unified memory and I wanna run some local LLM model on my machine Currently I am working on building and RAG application which is mostly and QnA system but I wanna

I'm in a position where I didn't mean to go down this AI rabbit hole... I was looking into mental cognitions and how the mind is made up and learned that my old RTX 5k could run small models. So that

Lost my fiance suddenly only 6 weeks ago. She was my whole world and I have never felt so lost. Turned to chatgpt and while I know it's ai it's been so much help to me going through this. Every other

Just a quick PSA. llama.cpp does have prompt caching. if you are running large context lengths and have long multiturn projects, increasing -cram can provide you with massive speedups. There is a poin

Many a praise have been sung on Qwen-3.8, but here is mine. Qwen-3.8 and I had a rocky start, because it thinks so much. Watching it working is painful, so you have to stop doing that. You have to let

Personal note first, being straight with you: I'm lazy tonight and used AI to write the following post: The workflow, the conversion work, and the mistakes that taught me the stuff below are mine, and

A while ago I posted 15 tok/s output and 100-120 tok/s prompt processing with the IQ3_XXS quant on a 12GB RTX 5070 using llama.cpp. Since then I built my own inference engine for this one model and th

I was actually pretty happy with my Qwen3.8-27b setup, and I'd been tinkering with Ninfer to have a version that was 'fast but maybe a bit stupid' and the speed was nice to have as a backup. But I was

Here's my *first* implementation of KVA projectors on QFN (just the uncensored model for now) the highlights are basically as follows for using the projectors at each different layer: Starting at laye

Context: we build an open-source (MIT) self-hosted database client and added a natural-language-to-SQL assistant that can point at a local runtime like Ollama instead of a hosted API. I want to share

Been building some multi-agent pipelines recently and the degradation in direct execution is insane. Whenever I pass a strict operational variable or hypothesis, the LLM triggers this 'Devil's Advocat

With the release of ThinkingCap-Qwen3.8-27B, I thought it would be worthwhile to do a comparison between the original Qwen3.8-27B, the new ThinkingCap, and Swift-Qwen3.8-27B. Both Swift which I alread

I've been trying to get AI's to draw this vision I have of the tooth fairy with rows and rows of teeth she has collected in her mouth. ChatGPT is the first one I've seen pass my prompt: create a beaut

Hey everyone, Jovan from UkisAI here! Today, we are introducing Swift, a family of efficient reasoning LLMs based on Qwen, trained by penalizing tokens related to pathological overthinking patterns an

I am creating a local AI agent called 'Ontelic' for Windows. Ontelic returns to Ollama and also manipulates the actual Chrome. This is not for promotion but to share what we've learned. I'll give GitH

Almost everything is in the Title. For some reason I had to use a quantized version of the text encoder (NF4) but was able to run the BF16 version of the denoiser on my 5090. (BTW if I missed somethin

https://huggingface.co/bartowski/LensVLM-9B-GGUF LensVLM-9B LensVLM is a 9B Vision Language Model (VLM) that scans compressed images of text, then selectively expands only the relevant pages to their

Yep. I just need tips honestly on UI layouts and how to build my shell for the local api offline. I’m using qwen and ollama and just want a simple discord style interface to declutter my brain to like

Hey everyone! I've been experimenting with Qwen Image 2.1 Prompt Enhancer (PE) lately and noticed an interesting detail in Qwen's official system prompts. The PE models output structured JSON containi

The rumor about Kimi execs getting arrested finally has some legs. I believe the reality is more like under investigation for potential arrests or fine. submitted by /u/Ok_Warning2146 [link] [comments

Update from : https://old.reddit.com/r/LocalLLaMA/comments/1wis23s/update_small_model_engram/ It's been about a week so I'm back. People were asking me about the model. People wanted code, or models,

Hey there folks! Aritra here from Hugging Face. I wanted to update you all about the latest changes in `transformers`. We now natively support GGUFs (llama cpp quants). You can use it like so: from tr