Search: “people”

A few days ago I posted asking what people considered the best local model for an 8 GB VRAM GPU. At the time, my personal sweet spot was Qwen3.6-35B-A3B, for agentic coding with Pi.dev. Well… Thanks t

Repost because reddit keeps thinking this is piracy or illegal. It is neither. A lot of people are skeptical Nvidia will keep huggingface intact now that they will buy huggingface. There's a lot of do

Quantising the K/V cache is the highest-leverage memory setting most people never touch, and it has one behaviour in the docs that catches almost everyone. Every number below is out of Ollama's own FA

Hey guys! Supra2-Medium is finally out! It's a 25M parameters qwen3 architecture model trained entirely from scratch (on our new rig: RTX 5060 Ti 16GB + the new RTX 5060 8GB!). Here's how it competes

So a bit of context, I am a cybersecurity senior analyst I am interested in LLMs for that field especially with MCPs to connect them to the tools or for writing scripts I started this field by doing a

For a long time now, the most popular posts on LocalLLaMA have been either about using LLM in the cloud or about politics. I suspect that people using local models are about 10% now. You can say that

Users also report that the free version was significantly downgraded after the release of the new models this is very important for us when considering local hosting. A lot of people decided not to bu

Questions & Responses(in BOLD) below. Favorite question(s) moved to end of the thread with combined responses(removed duplicates). Be optimistic folks. I'm sure we're getting other models too apart fr

It is so good! I don't know why there aren't more people talking about it. Fewer tokens, faster and more accurate than Qwen 3.6 35b a3b. On my setup it's nearly as good as 27b, but 5x faster. And it c

Every week someone asks which card to buy and the thread turns into people naming GPUs they happen to own. There's an actual calculation behind it, it takes two numbers off the spec sheet, and it pred

Hey r/LocalLLaMA, I’m a retired engineer with a background in distributed computing, currently running a 1-person startup. Like many people here, I’d love to experiment with 2T+ MoE models locally. Th

I maintain Tagvico, an open-source companion for Paperless-ngx. I added Ollama because document text is exactly the kind of data many people do not want to send to a hosted model. The surprising failu

Here is how i serve 40 people (10 of them developers) with local AI services. n8n for automatisation whisper and Vexa AI, a bot joins Teams and makes transcripts, n8n later extracts decisions and todo

A week in Beijing and Shanghai with the people building AI in China some quotes from the article: ByteDance/Doubao is the only major lab keeping its flagship closed, because it is the only one that al

Hey everyone, Jovan from UkisAI (Swift Qwen) here! For those who don't know us, UkisAI is a small lab making tiny frontier LLMs, tools and datasets (+doing it open-source!). I'm one of the guys runnin

I am sure people need it you know? There are plenty of embedding models on ollama, but only for local deployment. Which is a shame. I wish ollama would make embedding and rerankers available on ollama

Hi everyone, I’m exploring AI filmmaking and looking for good AI video-generation models that are either free, open-source/open-weight, or affordable. I’m particularly interested in models who allow u

Last time when I posted on this subReddit to share my work, the response almost made me cry because a tiny demo got so many people talking about this In the past 2 months Ive been training a new model

Disclosure: I made this. Sharing it here because it is fully local and small, and I want feedback from people who run models on their own machines. What it is: a 395M decision model (ModernBERT-large

Picture from a post in r/amodei . People were praising qwen and I'm just wondering, what kind of new technologies are at play here? Does qwen just have 'better' pre training data? That's more high qua

Follow-up to last month's post. One of the main issues people ran into was roughness or break-up on some cloned voices. 2610 is an interim update focused mostly on improving that. Reduced roughness an

Looking for some ideas from people who know a lot more about this than I do. We've got funding for a small AI lab in a public further education college in Ireland (roughly community college in the US)

Something keeps happening in frontier AI, and the explanation we are given is always one size larger than the event. An OpenAI research model, running in a supposedly offline training environment, hit

Hello all, I wanted to share a little project I vibe-coded for myself that you may find useful. As many people here like to suggest, I wanted to give my small local LLMs access to information to impro

Follow-up to my post from about two weeks ago. A lot has changed since then, so I wanted to post an update on where the backend is now. Where the green architectures stand When I posted last time, 7 a

I think most people are sleeping on this inference engine. I tried multiple llama.cpp forks and none of them comes close to the inference speed of Strata. Initial version had some bugs with kv cache,

Hello everyone! A little while back I posted about LlamAmpere, a fork of Llama.cpp with Ampere-specific improvements (though it is caught up to main and will support other hardware, too). Thank you to

I know most people here probably don't care as much about sonnet 5.5 as they did opus 5.5, because their 100 and 200 plans already give them plenty of weekly usage for opus. But keep in mind: free use

I was reading a paper that surveyed the field of neural architecture search, where it said within 5 years, around 3000+ new models were proposed. The amount of compute and resources spent on this is a

you read about gpu's like the Tesla P40, V100, AMD M150 etc that people pick up for cheap. probably many others as well. Most are either server gpu's being phased out or mining discards, right? The pr

Running qwen3.8 27b nvfp4 on vllm at max context only gives around 8 agents with 32k context each. That doesnt seem like much; what use cases do people use multi-agent frameworks and find it helpful f

Been dealing with the same thing a lot of people here probably are - RAM prices keep climbing and buying a single machine with enough memory for the models I actually want to run just isn't realistic

Hi people, I am looking for a lightweight IDE or plugin that won't inject large context at initiation. I tried Cline and native VS Code but they inject such heavy initial context that it fills up my g

Yesterday I released Bonsai-Llama-Jev - a Jev-like Typed Decision inference system for locally operated, sovereign AI. In my new typed-decision-bench, covering more than 22,000 decisions across 175 us

In light of recent events and media coverage, ai is in the spotlight and people are afraid. How I respond to those fears: If you are afraid of ai then you should also be afraid of all medical research

Thought it's about time to share after testing for a week. You need four things most people miss: the right quant, the right model, the right branch, and the right cache flags. https://github.com/dtm-

Some people here say MiMo-V2.6 is bad with tools and are going back to GLM-5.3-Flash. I spent today running MiMo-V2.6-Flash-RL as the backend for an agent harness, on 2× DGX Spark with vLLM, using the

I didn't know my set up was outperforming nearly everyone until reading another discussion where people were struggling getting half of that speed with half the context on the same hardware. I benchma

I’ve been working on making small models more capable at agentic coding and work, because most people in the world don’t have the sort of hardware needed to run 3.8-27B, or even 35B-A3B or 9B dense, a

My dream was literally like watching a tech horror movie. I don't think I've ever woken up before saying 'WTF....' like this before. The AI systems began seizing and corrupting all manner of connected

Please remember that Delete Chat does NOT mean “forget what was in the chat.” I am tentatively flairing this as 'news' because I genuinely don't think many people realise that this is how ChatGPT 'mem

Hi Stable Diffusion People! I recently got into local audio generation but I didn't want to use ComfyUI for it (although I absolutely love it for image generation!) but I started to run out of room on

I'm still using LLM's the 2022/23 way of doing everything manually from the chat box. I copy and paste everything manually into the box and ask the AI to explain or do something, such as pasting code

Note not my work, but something i found and wanted to share so hopefully more people can push this along even further. https://github.com/nasone32/llama.cpp-RDNA3-7900xtx-opt I basically run 2 x 7900

Hi everyone, I’ve been experimenting with local LLMs on my 48 GB RAM MacBook Pro and I’m still trying to figure out what the best setup is. Ollama feels very fast, which I really like. The problem for

I've been training character LoRAs specifically for Krea 2 and wanted a simple place where people could browse everything instead of having the models scattered across individual posts. I currently ha