All entries

Just a quick PSA. llama.cpp does have prompt caching. if you are running large context lengths and have long multiturn projects, increasing -cram can provide you with massive speedups. There is a poin

Many a praise have been sung on Qwen-3.8, but here is mine. Qwen-3.8 and I had a rocky start, because it thinks so much. Watching it working is painful, so you have to stop doing that. You have to let

Personal note first, being straight with you: I'm lazy tonight and used AI to write the following post: The workflow, the conversion work, and the mistakes that taught me the stuff below are mine, and

A while ago I posted 15 tok/s output and 100-120 tok/s prompt processing with the IQ3_XXS quant on a 12GB RTX 5070 using llama.cpp. Since then I built my own inference engine for this one model and th

I was actually pretty happy with my Qwen3.8-27b setup, and I'd been tinkering with Ninfer to have a version that was 'fast but maybe a bit stupid' and the speed was nice to have as a backup. But I was

Here's my *first* implementation of KVA projectors on QFN (just the uncensored model for now) the highlights are basically as follows for using the projectors at each different layer: Starting at laye

Context: we build an open-source (MIT) self-hosted database client and added a natural-language-to-SQL assistant that can point at a local runtime like Ollama instead of a hosted API. I want to share

Been building some multi-agent pipelines recently and the degradation in direct execution is insane. Whenever I pass a strict operational variable or hypothesis, the LLM triggers this 'Devil's Advocat

With the release of ThinkingCap-Qwen3.8-27B, I thought it would be worthwhile to do a comparison between the original Qwen3.8-27B, the new ThinkingCap, and Swift-Qwen3.8-27B. Both Swift which I alread

I've been trying to get AI's to draw this vision I have of the tooth fairy with rows and rows of teeth she has collected in her mouth. ChatGPT is the first one I've seen pass my prompt: create a beaut

Hey everyone, Jovan from UkisAI here! Today, we are introducing Swift, a family of efficient reasoning LLMs based on Qwen, trained by penalizing tokens related to pathological overthinking patterns an

I am creating a local AI agent called 'Ontelic' for Windows. Ontelic returns to Ollama and also manipulates the actual Chrome. This is not for promotion but to share what we've learned. I'll give GitH

Almost everything is in the Title. For some reason I had to use a quantized version of the text encoder (NF4) but was able to run the BF16 version of the denoiser on my 5090. (BTW if I missed somethin

https://huggingface.co/bartowski/LensVLM-9B-GGUF LensVLM-9B LensVLM is a 9B Vision Language Model (VLM) that scans compressed images of text, then selectively expands only the relevant pages to their

Yep. I just need tips honestly on UI layouts and how to build my shell for the local api offline. I’m using qwen and ollama and just want a simple discord style interface to declutter my brain to like

Hey everyone! I've been experimenting with Qwen Image 2.1 Prompt Enhancer (PE) lately and noticed an interesting detail in Qwen's official system prompts. The PE models output structured JSON containi

The rumor about Kimi execs getting arrested finally has some legs. I believe the reality is more like under investigation for potential arrests or fine. submitted by /u/Ok_Warning2146 [link] [comments

Update from : https://old.reddit.com/r/LocalLLaMA/comments/1wis23s/update_small_model_engram/ It's been about a week so I'm back. People were asking me about the model. People wanted code, or models,

Hey there folks! Aritra here from Hugging Face. I wanted to update you all about the latest changes in `transformers`. We now natively support GGUFs (llama cpp quants). You can use it like so: from tr

Original Post : https://www.reddit.com/r/StableDiffusion/comments/1wnhqm3/open_source_mingimage01design_family/ Kijai's PR : https://github.com/Comfy-Org/ComfyUI/pull/16482 NB: I gave practically zero

Hi, this is the research project I'm working on. I must say, building a GUI harness is way more complicated than I thought. The goal is something like Minecraft. You know, a few simple tools that a us

Yesterday I released Bonsai-Llama-Jev - a Jev-like Typed Decision inference system for locally operated, sovereign AI. In my new typed-decision-bench, covering more than 22,000 decisions across 175 us

I keep seeing Jev presented as some new class of decision model, but most of what’s being advertised is just normal classifier behavior with modern zero-shot capabilities. It outputs probabilities ove

I’m the developer of TensorSharp. Inspired by vLLM PR #57250, I added and optimized a native /v1/systemone endpoint for Jev-style structured decisions using DiffusionGemma GGUF. This uses DiffusionGem

I made a new Krea 2 LoRA called Anygles. Give it one image of a person, then spin the camera around them, move it higher or lower, zoom in or out, or mix everything into a random camera path. The four

Kijai put his magic to good use to improve your MiniMax H3 workflows. The video VAE now encodes up to ~2.2x faster and decodes ~1.4-2.7x faster on Nvidia GPU. Checkout the blog post how to use it: htt

In light of recent events and media coverage, ai is in the spotlight and people are afraid. How I respond to those fears: If you are afraid of ai then you should also be afraid of all medical research

MiMo-V2.6-Pro has an insanely high score of 46 on AA, putting it at the head of the opensource models available. It also costs pennies. Flash is not out on AA yet, but it costs less than half on datac

Looking for the best way to restore/refine quality of an AI-generated 720p video to 2K I’m looking for advice on the best ComfyUI workflow for improving the quality of an AI-generated video. My workfl

Worth a read if you track open weights. Nathan Lambert (interconnects) posted the written version of his Congressional testimony on where open models stand. The numbers that got me: Hugging Face downl

I’m looking for advice as I’ve just started out the local llm journey. I’d like to try out coding. I downloaded LM Studio and Qwen3.8-27B-Q4, but I wonder if there are more optimal options for my hard

Is it just me or ollama serve command doesn't work at all. It uses a different directory than the systemd process, so it can't find any model or anything if you installed it using systemd ollama. Luck

I have been using open-webui and open-terminal. Open-terminal is installed in a VM and gives the agent a ton of control of it's own system that I keep away from my normal network. I find this gives th

I am proud to release these quants of Qwen 3.8 27B. They beat the excellent ISTA and Unsloth quants byte-for-byte on three corpora. Both KLD and top 1% were tested 3x. It took a week of continuous GPU

If like me you enjoyed classics like Baldur's Gate 2, this is a small fun experiment. For anyone interested: https://www.youtube.com/live/8FhPfRKTucw?si=PMIVdHbZ-31PNwZD Qwen 3.8 Flash Next has amazin

Thought it's about time to share after testing for a week. You need four things most people miss: the right quant, the right model, the right branch, and the right cache flags. https://github.com/dtm-

Got QFN up and running on our Strix box this past weekend and have been running on it for a few days now. Big thank you/shout out to the Halogen team, it's running fantastic on the Strix, this is clea

First: That the 7GB (int8) 2.1 comes even close to the 20GB (int8) 2511 is amazing, but maybe because of the smaller size some capabilities got lost. For some tasks 2.1 it is much better than 2511, bu

Qi2.1 Viggle 0.2, 5steps, CFG2.1 prompt: Cute young woman at beach holding Grumpy Cat. We're still getting a soft focus with the 'baseline'steps of 5, non-baseline CFG 2.1.... Now she is wearing a swe

Hey everyone, I made a simple script called MOMOS to run Ollama and small LLMs directly on your Android phone using Termux. It automatically handles PRoot-Distro (Debian), installs Ollama, and lets yo

I’ve been playing with Nemotron 3 Diarization, and it fills a gap I’ve had with local voice agents: keeping track of who is speaking. It’s a diarization model, so it gives you speaker labels rather th

What it is. stuntd is a local proxy for typed LLM decisions: choice, yes/no, score. Your app keeps calling its provider through it. stuntd records the decisions, trains a small head on top of Laya (th

Illinois folks (like myself), write your governor, please. And if you live elsewhere, please pay attention to your local/state politics. https://wgntv.com/politics-3/pritzker-establishes-ai-cabinet-to

Saw a new open 27B model release that looks like it could be interesting for the Ollama crowd: VeriLoop E2. It is a Qwen3.8-27B post-train aimed at code, mathematics, scientific reasoning, and agentic

Hey everyone, I’m new to running AI models locally and I’d like some advice. My laptop specs are: HP OMEN 15 CPU: Intel i5-8300H RAM: 16 GB GPU: NVIDIA GTX 1050 Ti 4 GB VRAM OS: Windows I’m planning t

Hey everyone, I'm getting into running LLMs locally with Ollama and would like some recommendations on which models I should realistically run with my hardware. My PC specs GPU: RTX 5060 Ti 16GB VRAM