Search: “local-ai”

The models are fine, our toolings and methods are shit.

I've hit again a point where me as a developer have to take a break from all this slop shit. Im a Developer for 13+ years and i loved it. But i fell for the slop trap. First it started with copilot an

7 Sept 2026

Qwen3.8-27B 'Unhacked' my PC

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

DGX agent

Right, so this is going to be embarrassing but it's presumably something we've all been through at one point or another, and I guess this is my first time resolving something like this in the way that

local-air-localllama

6 Sept 2026

Villager Simulation Game POC Created with Qwen3.8-27B-UD-Q3_K_XL.gguf - 16GB VRAM

https://village-sim-one.vercel.app/ - 16GB VRAM RTX 5070 Ti, fully offloaded - Vision on CPU - Windows, not headless - beellama.cpp - latest version with the kvarn performance enhancements making it a

6 Sept 2026

My only real use case for a local AI use is document management, how much VRAM do I realistically need for a good experience?

I just want to use paperless-ai and be able to ask questions relative to it. Bonus points if I could use it with home assistant but that's not the focus. I just can't see needing a 32 GB VRAM GPU for

5 Sept 2026

Reusing old hardware for starting local ai-journey?

Hi, I am thinking about putting some money / time into my local ai learning path and 'cleared' my attic where I found the following hardware. 3 x NUC11 (Core i5 1145G7 / 2,6 GHz, 64 GB RAM DDR4 2666MH

5 Sept 2026

Title: MLX-DLSS: NVIDIA's DLSS 5 neural renderer and frame generator, running on a Mac (and in PyTorch anywhere)

NVIDIA ships two neural networks inside DLSS: the DLSS 5 renderer that makes game frames look more photoreal, and the frame generator. Both only run on RTX cards inside games that call DLSS. I pulled

5 Sept 2026

Help me understand gguf size/ctx size

Let's say I have 2x 16Gb GPUs and I want to run Qwen3.8 27B. Monitor is ran by the integrated GPU so both 16Gb GPUs are almost fully free. I load the UD-Q4_K_S on one card at 15.4Gb. I then load the c

How do you guys handle your personal RAG setup

I am getting into developing a RAG setup, for getting information out of existing documents, new document ingestion, web searches, and good visuals. I am planning to use it for, alongside the regular

Qwen3.8-27b is the first Local model im able to blindly trust

You know that thing where you just throw a task at a frontier model and not have to supervise it worrying of it going off course? Qwen3.8-27b has officially gotten me to that point for local work. He

Tossed distorted audio samples to an open-weight voice model; it did fairly well.

Being a person obsessed with testing new models that come out, times are really insane for me. Tested different kinds of TTS and voice cloning models but none of them gets it right in terms of emotion

You can now run a 90M conversational LLM on the Sony PSP (hardware from 2004). Doesn't get more local than this.

Github link: https://github.com/thatblend/LLMPSP I wanted to see what the PSP can theoretically handle and I got my answer - a 90M model is about the max it can do without atrocious inference speeds.

Best small LLMs for translations?

I am working with multi-language content and have so far been useing gemma4:e2b for general work. Today I tried it with Russian for the first time and the result was not great. Gemma4:e4b was better,

Can a 4B local model actually feel like an AI assistant?

I've been building Arcon around Qwen3-4B + LoRA. Instead of just making it a chatbot, I'm experimenting with persistent memory, personality/mood, internal state, tools, and eventually having it proces

First results from H3 Acceleration Arena

https://preview.redd.it/c8za3q7shanh1.png?width=900&format=png&auto=webp&s=5716c30be6913c351403fb16cc60a59e3b116200 https://huggingface.co/spaces/multimodalart/h3-acceleration-arena From author u/apol

Frontier models sabotaging local AI implementations?

For a few days I've been working on creating a custom local-only harness for some work related research using Codex / GPT 5.6 Sol and the model feels not only dumber than usual, but straight up counte

I do not wish for Nvidia to have a majority stake in Hugging face.

I like Nvidia, Ollama, and Hugging Face. I wish each of them well. But one owning the other creates a huge conflict of interest. Separation of Powers is key for long term sustainability of open weight

Looking for Captain Obvious

I might be missing something obvious, but when I open any cloud model, by default I'm dealing with the 200k version and the first thing I need to do is /model to change it to the 1M version. Is there

Multiplayer world simulations with h3 max

Discovered that using a realtime llm + h3 world simulation, you can allow multiple different simultaneous character instructions that are restricted to only controlling their respective character. If

New ollama plan usage

Bruh wtf, i switched to new method now im pissed, 20 dollars a month for 60 api cost a month ?? Open code provides this for 10 dollars a month , and if I use a beast like kimi k3 I can drown my months

An endless AI TV channel on a single gaming GPU — MiniMax H3, generating faster than it plays

There is a video stream running on my desktop right now. It has sound, it has never repeated itself, and it will not stop. I point VLC at a local URL and it plays. One RTX 5090 does all of it — no clo

LocalLLaMA is unironically one of the best places to go to get up to date AI news.

One of the other posts today by user u/Howard_banister confirmed what I've been seeing from the other AI subreddits as well. Most of these other subs are 90% trend hopping crypto-bros equivalent peopl

Old Pro plan isn't the same anymore

I'm still with the old pro plan, and I've never (almost) reached my weekly limit so fast like today with that many requests and using OpenClaw for pretty basic stuff. There is some serious monkey busi

Qwen3.8 Flash AP Quants

Quite surprised to be beating other high quality quants. It took a lot of benchmarking to get here and we are quite pleased with these, hope they are useful to the community. It required a modified wa

A very confusing report from Puget Systems

Just to name a few: running Qwen3 8B on a 32GB GPU running Qwen3.6-27B Q4_K_M on 2 x R9700 quote: 'each prompt was sized at 500 input and 500 output tokens' for a full system that costs $18,775?? I do

1 Sept 2026

I pushed Qwen3.8-27B to 2.000 prefill per second and 132 decode per second on A RTX 3090.

Yoyo I'm back with updates to the fastest inference engine with minimal quality loss for Qwen3.8-27B. The last few weeks I've been optimizing decode speed and I don't think it can be pushed further, u

1 Sept 2026

How bad do you think models like Qwen3.8-27B or GLM-5.3-Flash would be with H-Neurons disabled?

TL;DR: this paper proposes a method to fix hallucination rates to very low levels or zero by disabling neurons which contribute to hallucination. This discovery has been out for a while now, but it ha

Optimal 1.25 bit quantization of Qwen3.8-Flash-Next

Hello! I was looking into quantizing models and i saw how Hy4 was shrunk from 1.5 TB to 200GB with high retention in benchmarks (98% i think). I was wondering if: a) it would be worth it to attempt th

Sliding-window beats linear attention

Interesting new paper from Alexia Jolicoeur-Martineau (of Tiny Recursive Model fame) and collaborators. They seem to be able to replace quadratic attention with sliding window attention + attention si

SlopTV: an infinite livestream of AI slop generated from youtube chat comments, Minimax H3 on 2x5090

SlopTV: a YouTube live stream where the chat writes the programming. You type 'capybara dj underwater rave', an LLM inflates it into a 400-word structured video prompt, one of my 5090s renders 15 seco

Are there any interesting architectural innovations that we seem to be on the verge of for LLM models or AI models that might be a big deal? (Excluding maybe N-gram, since everyone is already well aware of that one)

So, ideally for this thread we exclude the ones that everyone on here is already well aware of and discussing on here a lot, like N-gram, quantization improvements, MTP, D-flash, and D-spark, since th

Got MiniMax H3 video generation running in TensorSharp

I’ve been experimenting with MiniMax H3 and finally have video generation working in TensorSharp. TensorSharp started primarily as a local GGUF/LLM inference engine, so getting a video-generation pipe

Is it possible to run DLSS 5 on RTX 3000-series GPUs with an INT8 conversion?

I see this kind of approach quite often with local AI models. For example, when optimizing models for an RTX 3090, people sometimes convert them from FP8 to INT8 so they can run efficiently on Ampere

Qwen3.8-Flash-Next at 170K context on a single 96 GB card. ~110 tok/s.

I used the quantized n-gram to INT4, it's 32 GB, memory-mapped from disk. I confirmed that it works great on 150-160k context, and i was watching all the time my VRAM usage while doing single thread l

Qwen3.8-Flash-Next NVFP4 2xDGX Spark config: 50t/s decode, 2,900t/s prefill

After a couple of days of faffing around, here's my current config in case it helps anyone out. Some of this is also valuable for a single Spark. AI slop below for you to read and paste into your own

Qwen3.8-Flash-Next turns 4xR9700 into a local AI powerhouse! 120 t/s TG and 12k t/s PP single request with optimized vLLM

If you own 4xR9700 and were waiting for the model to make them shine, then I have some good news for you! It's running at 80-120 tokens/second for generation and 12k token/second prefill for a single

AutoGPT from source with local models: complete setup guide

I put together a technical guide for installing AutoGPT from source, with a focus on using local models instead of OpenAI. It includes Docker Compose files, environment variables, and how to point the

Creative destruction

This has been a big 2 weeks for local models with 3.8 27B and Next coming out, a lot of 'frontier labs are done' comments which, in general, could certainly be true. Could be trillions in economic val

Did anyone else notice the Ornith 1.5 35B GGUFs got a 'silent' update?

Hey all, I stumbled onto something funny and figured I'd share before it gets buried. I'd been running the Ornith 1.5 35B A3B GGUF (Q4_K_M) for a while, then recently a new revision showed up in my HF

Tenstorrent Qwen3.7-27b Benchmarks

I saw someone here posted about getting a Tenstorrent QuietBox 2, and I wanted to look into it the hardware. It's very difficult to find any benchmarks, but I managed to find some from an employee. Th

TestFlight Testers Needed! PowerChat – an AI chat client with RAG, MCP

Hi r/ollama! I am the solo dev of PowerChat, an AI chat client that provides a mobile chat interface for iOS (iPhones for now, iPad and Mac planned) to connect to self-hosted models using Ollama, or O

Were designing a tiny autonomous research agent

This base model is only 43m parameters trained on 3m arXiv abstracts. We plan to continue pre-training and post training. If you create fine-tuning datasets or if you know of any datasets that can hel

Computer Hardware to run on premises

We are considering to buy computer, server to run decent model on premises. I would like to run big open source model with more that 70b parameters roughly. I read the people used to run on Apple Stud

H3 - anatomical slider

Happy Friday! Ever had issues getting the anatomy right on your t2v generation? Just add a slider with your H3 Prompt! What have you all been building on your local AI studios? 384x448, int8, 20 steps

How do get refund

Has anyone been able to get refund from Ollama cloud? What is the process to get it. There are cheaper alternative now and I was stupid to pay for a full year cos it seemed lucrative then. submitted b

How sparse is too sparse for H3?

So me and various other people have implemented their own Sparse Attention nodes and you can see many people argue about what % you should actually run these nodes on to maintain prompt adherence etc.

I fine-tuned and published my own 8B coding model — SaphraFgt 1.0

I fine-tuned and published SaphraFgt 1.0, an 8B coding model focused on HTML, CSS, JavaScript, frontend development, UI/UX, code completion and debugging. Ollama: [https://ollama.com/SaphraFgt/saphraf

Row-Bot v4.9.0 is available

Row-Bot v4.9.0 is available. - Meet Buddy: a native, always-on-top desktop overlay for Windows and macOS. - Drag Buddy from the sidebar and place it over any app. - Chat, track progress, read replies,

Self-hosted agent + BI platform that runs fully local, Ollama or vLLM for both chat and embeddings

Been building AgentSwarms for a while. Agent platform with RAG, multi-agent workflows and a BI layer over your own data. Posting here because the whole thing runs without a single hosted API if you wa