All entries

Given OpenRouter.ai was snapped up by Stripe, who do we think would go after the 'GitHib' of AI models? It is a big chunk of change they are looking ($13B). Apple may be a contender to give them a rea

been building ShardFlow for the past few months, a distributed LLM inference framework that splits any HuggingFace transformer across N GPU machines and uses neural speculative decoding to deal with W

A few weeks ago I shared Agent Quest, my open-source experiment that turns Claude Code and Codex sessions into heroes living inside a small 2D world. The original idea was mainly about making it easie

WARNING: I think I made a mistake with omlx, and its results below seem to be without any kind of acceleration. I am rerunning them with Lightning MTP and ANE support for prefill. Will update the post

I haven't had a lot of luck in the reliability of the output of any models so far for local coding use, and find myself relying on Claude's limited free sessions to assist with coding. My coding appli

Hey r/Ollama, If you run local models on mid-range hardware, you probably know how fast VRAM disappears when you try to set up document memory or local RAG pipelines. I built Hillock (https://github.c

I haven't seen a post about this here, and I'm curious what you think about it. On August 14 Civitai rolled out the option for creators to choose 'permanent paid access - selling with no time cap'. Pr

This is something I've noticed. Back when GLM 5.2 and Kimi K3 launched, there was a media push on pushing how dangerous open source models are. We know why they were doing this; these open models are

I don't know about you guys, but Deep-seek harness is insane. It's not focused on being a coder agent, it's webUI made it very easy to just checkin from time to time, and the best part? Why it's bette

This is one more vibe slopped custom optimization for, in this case, my hardware (m2 ultra 60 cores, 192gb). It is just a fork from llama.cpp with a few changes, it achieves: - DeepSeek V4 Flash, no k

I've tried a few Gemma 4 models, for example 'gemma4:26b-mlx'. It says 'I cannot process images.'. On their website it mentions images, audio etc. https://ollama.com/library/gemma4/tags I’m using Olla

I'm running Qwen 3.8 27B locally on a single GPU. I can push the context to 131k, but I'd rather run it faster at 64k if the agent can manage context properly. What I have in mind is pretty simple: on

I'm building a local AI companion and I'm currently working on its cognitive layer. The goal is: User message → understand intent → decide what context is relevant → retrieve only useful memories/stat

Gemma 12B is obviously a very well trained model, I always thought the fine tuning they did on it wasn't really cut out for agentic coding. From my own experiences it struggles to use the tools it's g

What I ran: 8x B300 on Modal, 56.79 per hour, vLLM, tensor parallel 8, native MXFP4 Cold boot ~27 min (1.56 TB load, JIT, 51 CUDA graph captures) TTFT 0.92 to 1.02 s, decode 92 tok/s steady, 83 tok/s

The task was the same both times: replace 63 calls to datetime.utcnow() across 22 files in a Python service, keep the comparisons that assume naive datetimes working, and leave the tests green. I read

My setup: RX 6700 XT 12GB i5-11400F 16GB RAM Ubuntu 26.04 ROCm/TheRock + llama.cpp I've spent the last few months experimenting with different backends, quantization, CPU/GPU split, KV cache, batch si

I recently implemented a minimal, educational version of SynthID-Text-style watermarking for language models. I saw anthropic post about how they'll start adding watermarks to their model responses an

Tap is Linux dictation: hold Ctrl+Space, speak, text lands in the focused app. Local, on-device.(Cloud Option Available) Built in public. What are some features you guys would like to see, before the

Hi all Looking at a PowerColor R9700 for Qwen3.8-27B, Q4, llama.cpp/Vulkan. AMD's own blog quotes 51.8 tok/s but doesn't say what context length that's at, or whether MTP=2 was holding up. Separately

Hey everyone! Last year, Mozilla released Orbit, an AI-powered browser summarizer hosted on a GCP server. After people started digging into the extension, they discovered things like backend endpoints

Hey everyone! Last year, Mozilla released Orbit, an AI-powered browser summarizer hosted on a GCP server. After people started digging into the extension, they discovered things like backend endpoints

i got it last week and im really impressed. its maybe like a sonnet from last year or 6 months ago. i had built some automation for myself and a dashboard cobbled together over the years from bits of

I was just curious how the new qwen3.8:27b does on a hard C to HTML porting job against Opus 5 in a default Claude Code. The job: my fun side project is a procedural shooter in a single C file. Port i

I noticed that a recent Ollama/llama.cpp setup was using a surprising amount of CPU despite the model being fully GPU-offloaded. My setup: Ryzen 9 9950X VM with 16 vCPUs 3× RTX 4060 Ti 16GB Qwen 3.8 2

ollama max subscription is not accepting new users since july 27th. https://web.archive.org/web/20260727111430id_/https://ollama.com/pricing . is there any prediction on when will they allow upgrading

Most YouTube benchmarks only show trivial tasks like generating landing pages or simple Three.js games. Is a local model like Qwen 3.8 27B actually capable of real-world systems programming—such as bu

Our devs got their hands on it a few days ago. One wired it into Codex to compare with GPT Luna, our usual workhorse right now for its cost effectiveness. Another tried it out on one of our OCR pipeli

I was fascinated by Nanbeige's outstanding performance for its size, so I started digging into how much a model can improve its own representation just by looping over itself (for fun). My prototype w

I have been lurking on reddit for like 6 years, and this is the first time posting, so apologies for the mistakes. I'm not a native English speaker, so sorry for the bad English too. I used LLM to tra

Hello, So I've been trying lots of combinations in that never-ending landscape of options and settings. I wanted a proper quant of 3.8 27B running as fast as possible on my 5090 at 400W, with vision a

Further to my last post, https://www.reddit.com/r/LocalLLaMA/comments/1vldngi/tested_in_coding_bf16_muse_glimmer_vs_bf16_qwen36/, I bring this requested comparison of Qwen3.8 27B. Context: Both models

Earlier this year I posted about building what at the time I believe was the first 16x DGX Spark Cluster. I’m now adding 20 more Sparks to the cluster in my homelab server rack, giving me 4.6TB of uni

Hi everyone, I'm currently working on optimizing an LLM setup at my company and I'm trying to identify where the actual performance bottleneck is. Our current setup is roughly: LLM: NVIDIA Nemotron mo

https://huggingface.co/jojohai/Qwen3.8-27B-MTP-graft Tested on Vulkan, the grafting saves RAM compared to using an external file. What I don't guarantee however is the quality of answers. The model is

Me and my team made Atomic Dynamic GGUF quants for Qwen 3.8 27B, so we wanted to see the difference between them by giving each quant the same voxel island creation task First of all we were surprised

It performs at full capacity without any issues anywhere, but only the Ollama DeepSeek seems particularly trashy. Why on earth has the Ollama DeepSeek unusable trash that goes into over reasoning and

If anyone still remembers GLM-4.5-Air from last year, you can now get a nice speedup by enabling MTP in llama.cpp. It is a 106B MoE with only 12B active parameters, which makes it interesting for mach

Spent way too long confused about why an 8B at Q4_K_M, about 4.9GB on disk, would OOM on a 12GB card. The weights fit twice over. The answer is the KV cache, and the math is simple enough that you can

What models and configs are we using? Please share here On windows, I am using this copium pared down model https://huggingface.co/Bucoid/Qwen3.8-27B-Uncensored-IQ4-XS-MTP-16GB-VRAM-GGUF with MTP disa

The post describes some experiments I had while trying to desperately run deepseek-v4-flash-0731 4 bit+ quants on my machine which is supposed to support only q2 quants of the model, a or 2.xx bpw qua

I've been working on a project for a few days and now all of a sudden chagpt is unusable.. It will list the next steps of implementation but never actually start working on them. anyone else? UPDATE I

https://preview.redd.it/84zi5nsdawkh1.png?width=2368&format=png&auto=webp&s=1109e69db807b153064b1f5b61d22cf1e9fbca05 Another user posted the benchmarks for Qwen 3.8 27B today, and while I think Qwen 2

My hobby project skill-issue is an arena FPS in a single C file, about 34k lines, everything procedural. My test for local setups is to have an agent port it to a single file HTML version with three.j

Context: Earlier, open-source large models like Mimo V2.5 Pro, DeepSeek V4 Pro (first version), and Kimi K2.5 used to struggle with this prompt, and Qwen 3.6 27B couldn't even render the globe properl

Is there a good rule-of-thumb to calculate how many full 5-hour windows add up to one week's allowed usage under the $20/Pro plan? ie: you fill up six 5-hour windows, and that equals 100% usage for a