Search: “llm”

I know cloud subscription plans are not exactly the core focus of r/LocalLLaMA, so I want to be clear about why I’m posting this here. I’ve been building my own local text-based RPG/game harness and u

Hey everyone, Like a lot of people here, I’ve been experimenting with running models locally through Ollama. Athena started as a small experiment, but it quickly grew into something more, a modular Py

I am trying to learn more about long-term memory and how people are handling that nowadays. Historically, I have used a mixture of paid cloud AI services (ChatGPT, Gemini, OpenRouter API) and my own l

Just find it interesting, since if Taalas tried to get into the consumer market, it could be interesting. Kind of like how we have game discs on CDs. Of course, a Qwen3.8-27B chip is stuck on 3.8-27B,

Hi just wondering has anyone submitted anything to JMLR before? Especially in the last 2 years? What is the experience like? Background: I am a Comp Sci PhD student, but my secondary supervisor (the o

Been building this for a few months, mostly for myself, and it just got a proper release so figured I'd post it. It's a native GGUF inference runtime with OpenAI/Anthropic-compatible APIs and a chat U

UPDATE https://github.com/ggml-org/llama.cpp/pull/29009 MERGED More Gen AI coverage and frameworks integrations to minimize code changes New models supported: On CPU: Gemma-3n On CPU, GPU: Kokoro-82M,

Another question asked 1 thousand time, but I can't find actual threads. Sorry in case of mistake. Due to interstellar price of hardware like high-end GPU (more than 5k euros for a 5090 and similar pr

After two days of fighting with benchmarking infrastructure, I finally benchmarked IFM/K2-Horizon-7B on 16GB VRAM. TL/DR: it works, but far behind Qwen-3.8-27B. Setup I compared 3 models - Qwen3.8-27B

I reallise this is not exactly a locally hostable model, at least definitly not on my hardware. But I figured some of you might have some insight on this. I run an agentic coding benchmark. The model

My prompt is: Epic high-fantasy concept art, ultra-detailed. A road crossing golden plains leading toward a colossal dwarven mountain hall entrance recessed into themouth of a giant cave. The whole en

Not sure if anyone out there is working on this, don't see any on huggingface to try out. It might be too new at the moment, but it would be cool to see the difference between a tiny model trained wit

Hey everyone, Giving AI agents access to MCP tools or shell commands is awesome, but prompt injection is still a huge headache. If an agent reads an untrusted document containing `ignore previous inst

So, uh... the popularity of so-called humanlike Qwen (currently on top in this sub) made me realize just how clueless the general public is about the models they have. You'd be shocked but you don't n

I wanted to see how my fully local home voice assistant compared to the latest GPT Live, so I tested it using the same conversation used in their 'Improved Intelligence' demo. In this video they ask t

so i saw that openui.com released OUI-1, a model fine-tuned on DiffusionGemma. the training dataset uses OpenUI-Lang, a custom DSL (domain-specific language), instead of plain HTML, Markdown, or React

Its chat template has this line: {%- set resolved_reasoning_effort = reasoning_effort|default('xhigh') %} xhigh is the most expensive of its three settings (low / medium / xhigh). If you never set one

Model available here: Qwen3.8-27B-Uncensored-Genesis-V1-MTP-GGUF This model is a practical realisation of things described in this paper, but adapted by me for machine learning: https://arxiv.org/pdf/

Most of the new ~30B (and now ~10B, thankfully for my GPU) models we see score really high on benchmarks, but I feel like they don't push back on dumb ideas enough. I think most people don't being lik

I found a lot of room on the table for these cards so I decided to make a specialized build to squeeze all I could. first The results: qwen 3.8 next Q3_K_XL: 920tk/s pp8192 (2 cards, ram offload), 24/

Gemma 4 12B runs on an RTX PRO 4500 Blackwell. Gemma 4 E2B run on a Jetson Orin NX 16GB; similar performance is expected on a Jetson Orin Nano Super 8GB. Both systems use a reSpeaker Flex 4-mic array

Quick setup on linux: STEP 0: install/download llama.cpp, Freecad, your favourite gguf model - possibly with multimedia image reading capabilities (I've used Qwen3.8-27B-UD-Q4_K_M and relative mmproj-

Hi all! I just wanna say that I am tired lol. Yes, it's another harness, but I spent a lot of time and effort and have forsaken my hobbies to build the Jenny (like XJ-9) app. Jenny is a free, MIT lice

Before the introduction, two caveats. This isn't anything as spectacular as the mmh3 stuff filling these subreddits these days, so please go easy on me. It's an experimental tool I'm sharing in its ra

I've hit again a point where me as a developer have to take a break from all this slop shit. Im a Developer for 13+ years and i loved it. But i fell for the slop trap. First it started with copilot an

This comparison was requested by a few people, and certainly we were all eager to see the final results. The comparison had taken 11 days and the GPU was crunching numbers for ~167 hours. We've been c

Right, so this is going to be embarrassing but it's presumably something we've all been through at one point or another, and I guess this is my first time resolving something like this in the way that

I ran a side-by-side ML text-processing and model-training workflow using Fable 5.1 vs. Astra (both on xhigh), and the results could not have been more different. Warning, long post. TL;DR -- Astra co

I just spent the last 2 hours of my life on a Friday night debugging a strange error in a prod CLI app. EF core was receive a readonlyspan during a Contains query. Normally, this query converted to a

I am getting into developing a RAG setup, for getting information out of existing documents, new document ingestion, web searches, and good visuals. I am planning to use it for, alongside the regular

If you are like me and use Google flow for NB2/NBP there is hope to finally ditch it. Let me start off by saying I’m not very impressed with any of the current local image edit or Image Reference mode

Disclaimer: I'm the builder. --- Over the past 2 yrs, I was working on a SaaS ML project & got very interested in ML/DL/AI. As everybody else, there were some normal paths I took to build a solid unde

I've made a modification of llama.cpp MTP for people that want to run models like QWEN 27B on 16GB and similar setup, the focus is reducing the memory cost of MTP allowing more speed for less ctx cost

TL;DR: this paper proposes a method to fix hallucination rates to very low levels or zero by disabling neurons which contribute to hallucination. This discovery has been out for a while now, but it ha

I used Claude Code to help write this pipeline - Gemma4 to turn an idea into a short story, then IndexTTS 2.5 running on an NVIDIA 3080 to use voices from LibreVox and VTCK voices to pin as character

I’ve been experimenting with MiniMax H3 and finally have video generation working in TensorSharp. TensorSharp started primarily as a local GGUF/LLM inference engine, so getting a video-generation pipe

Hey everyone, I am the co-founder of Tontaube. My brother and I just released TontaubeV1, a 2.9B-parameter open-weight TTS model focused on expressive speech, long-form generation, and low-latency loc

J'utilise souvent MiniMax H3 avec des images de référence ; je voulais essayer une vidéo que j'avais créée avec Wan (la version plus ancienne), mais cette fois en utilisant seulement une invite (un LL

Since Qwen's dropped the Qwen4Exp architecture bomb that focus on offloading parameters to n-gram instead of pure mixture of experts, I dug into this and learned quite a lot. Here's the summary. Expec

We're releasing a fully quantized NVFP4 version of Qwen3.8-27B. The checkpoint was trained using quantization-aware distillation (QAD) with QUASAR, our new QAT algorithm. We used the original BF16 mod

Hi everyone, it's been a while since I posted so here's an update on what the Lemonade community has been up to this summer. Our overall mission is to enable local AI builders with everything they nee

One of the things I like most about Ollama is how low it makes the barrier to experimenting with local models. The downside is that once you move away from high-end hardware, finding the right informa

A month i setup a proxmox server using spare parts like a z270 i7 7700k and 2x8 2666 but since i don't have any hardware acceleration i was planning to pass down my rx 580 8GB to my server but i heard

Minimax just announced (only over xhitter apparently) that it is raising rates by 60%-65% tomorrow, the 25th. If you use Minimax, you'll either want to lock in their current rates. Sadly, every chines

For the GPU poor among us: I'm curious what results you're getting with low quants of Qwen 27B 3.8. My main inference hardware is limited (Mac mini M4 24 GB), but I'm getting great results with Unslot

With vision enabled I am able to run at 150k context on a single RTX 3090 and the results are just amazing. I was even able to write a gmail plugin for DeepSeek harness with locally hosted Qwen 3.8 27