Search: “inference”

There is a video stream running on my desktop right now. It has sound, it has never repeated itself, and it will not stop. I point VLC at a local URL and it plays. One RTX 5090 does all of it — no clo

This may be somewhat specific to Qwen3.8 27b and the Apple M5 series, perhaps, but enough of us are running this combo that it's worth tossing out there. GGUF models with MTP have been the fastest way

I posted a new gfx906 based llama.cpp fork a few days ago. One of the main points of critique was that i did not provide sufficient numbers for the gains to be achieved. -- TL;DR: After switching our

I used a famous Simon Willison's pelican riding a bicycle prompt on the biggest local LLMs that can run on 128GB Apple Silicon. U used quantizations by Unsloth. Qwen3.8 Flash-Next gives a lot of detai

I ran a few benchmarks on my MacBook Pro M5 Max with oMLX: Running Qwen3.8-27B with thinking off has a big effect on output quality. Running it with thinking on xhigh burns 5.5x the tokens and runs 6x

hi i make tool for this called llmog it's purpose to make llms free to - auto annotation datasets - reclassification existing yolo datasets running totally local using llama cpp or vllm or use externa

Hey all! I'm finally doing some cool stuff with my 'thinking heater' (h/t u/-TV-Stand-). I'm still experimenting with GLM-5.2 (in anticipation of 5.3 coming tomorrow, I hope!) and things are very cool

I’ve been going back and forth on this for a week and I can’t settle it, so I’m hoping someone here has hands-on numbers. The two configs (German prices, dealer quote, incl. VAT): Config Price Mac Stu

My test is running DeepSeek v4 Flash 0731 at native size on DGX Spark machine (GB10, 128 GB unified memory). The model size is bigger than RAM, so weights will be loaded many times when running. To im

I noticed that a recent Ollama/llama.cpp setup was using a surprising amount of CPU despite the model being fully GPU-offloaded. My setup: Ryzen 9 9950X VM with 16 vCPUs 3× RTX 4060 Ti 16GB Qwen 3.8 2

Using llama.cpp I seem to be unable to get my to GPUs working tougether correclty, so I need help somehow. Setup: 96GB RAM, one Blackwell 5000 (48GB) and one 3090 (24GB). I am trying to run the UD-Q3_

This post was made with AI. I tried to remove as much slop as possible and keep it straight to the point to save your time as I know how annoying AI slop posts can be, but I still wanted to retain all

I've been experimenting with Qwen3.8-27B using DeepSeek Harness. It's a monster at long-horizon tasks, and the results were pretty wild. DeepSeek Harness ran on my Windows PC and connected over LAN to

I've got a dedicated AI rig sitting here with a RTX 5090 and 96GB RAM and for the past few years have been using Windows 11 and primarily LM Studio, but have also used vLLM, llama.cpp and Ollama. With

Including the rationalisation for the data below - this is a more robust version of an earlier post I did similar to this - explaining below: How I chose the comparisons The basic question I’m trying

Hey Folks, I've been making quants for a while - recently I took a short break to get into hardcore research (submitted my first EMNLP paper during it!). Along the way, I built up a little arsenal of

Hi folks, I hate slop as much as you do, so instead of starting with 'The Problem', I'll just cut to the chase: I just published Lophius, which is the culmination of more than two years of fighting wi

Basically what the title says. For me, it would always crash and burn trying to use tensor split. Apparently, there's some bug where GPU memory gets corrupted with the default microbatch (512) or high

Build a 100% offline fast Retrieval Augmented Generation (RAG) system that runs without an internet connection, without cloud APIs, without OpenAI/Ollama Published a video where you can build a fully

📝 Introduction We present Wan-Animate-2, a novel end-to-end character animation framework that directly consumes driving videos in a redesigned Diffusion Transformer, which achieves high-fidelity moti

Following up on yesterday's post about running everyone's faves on 2 x 16gb cards while maximizing performance and KV. Previous post data used abandoned Cu130 VLLM image. Stats here are done on cu129-

We run a local model instance in our company that the dev we hired built for us. We're a trade business and we want to further use our on hand hardware for it. The specs given we have is a 5090 gpu wi

Just wanted to share my agentic coding benchmark run of DSv4F 0731 at both High and Low reasoning efforts (not Max)... I ran a 109-question subset of Aider Polyglot (the JS/C++/Python languages), base

I have been working on this tool for months and there are a lot of new functionalities and tests that are going to be released in the next few weeks! The goal of the tool is to allow community members

I love the progress happening on open models but I feel like it is kind of getting clear that hardware to run good sized models is completely unaffordable for me right now. I know that you all love Qw

Hi all, I'm happy to announce that Xberg v1 is out. Xberg is the successor to Kreuzberg, equivalent to what would have been Kreuzberg v5. It's a content intelligence framework that handles a very wide

Hello fellow local AI people! I took 'you must create your own benchmarks' literally, and built a website for this. How does the end result look like Let's say I want to know which model has most comm

Mage-VL is a codec-native, proactive-streaming multimodal foundation model for image and video understanding, whose visual encoder is trained entirely from scratch at a compact 4B scale. It targets a

You patch security holes by intentionally finding them. If the models refuse to do it, how can companies protect themselves against rogue AIs, whether they are Chinese or OpenAI/Anthropic themselves?

I was refreshing my youtube and found out my favourite reviewer uploaded a battery test of 78 smartphones: https://youtu.be/MpgUFrsIWSQ the author said they started using robotic arm to simulate a per

There is a lot of buzz around running 100B parameter models on cheap local hardware using ternary (1.58-bit) quantization like Microsoft's BitNet architecture. The theoretical hardware shortcuts are i

At Bitwize, we've been building Logue, a native macOS app for AI meeting notes and writing, and we just open-sourced it (MIT). We're sharing it here because the whole point is that it runs 100% on-dev

Since a lot more people are trying to build their own multi-GPU machines, I thought I should help to prevent a common mistake people make with building multi-GPU machines. Which is using an Intel cons

Model quantization has become essential for efficient large language model deployment, yet existing approaches involve clear trade-offs: methods such as GPTQ and AWQ achieve practical compression but

TLDR: I (with the help of AI) re-implemented every Blackwell-only kernel (DeepGEMM, FlashInfer sparse-MLA, block-scaled FP8) in Triton, because they simply don't exist for sm89. The performance is 2-3

Disclosure: I work at Pluralis Research, the lab that built this. Code is open, and I'm happy to answer questions. TL;DR: As far as we can tell, this is the first RL post-training run whose entire rol

I'd like to search for information about this specific Reddit post to provide you with an accurate summary based on actual content rather than assumptions. SenseNova-U1 is SenseTime's unified multimod

Qwen3.6 27B is a 27-billion parameter language model that can achieve approximately 60 tokens per second throughput when running on dual RTX 5060 Ti GPUs with 16GB memory each, using the vLLM inferenc

Mooncake is the serving platform for Kimi developed by Moonshot AI, featuring a KVCache-centric disaggregated architecture that separates prefill and decoding clusters while leveraging underutilized C

Qwen3 is a series of large language models spanning both dense and Mixture-of-Experts (MoE) architectures, with parameter scales ranging from 0.6B to 235B, and a key innovation being the integration o

This Reddit thread on r/ollama discusses community questions around whether Ollama discontinued or changed its free cloud tier. Ollama Cloud launched in preview in September 2025 with fixed-price subs

This Reddit post from the r/ollama community discusses running local AI/LLM workloads using the NVIDIA RTX PRO 4000 Blackwell GPU via Ollama, a framework for running large language models locally. The

This r/ollama post describes a technique where an LLM is used to automatically tune its own llama.cpp runtime flags — such as parameters related to GPU offloading, KV cache quantization, batch sizes,

This r/MachineLearning discussion centers on Google's TurboQuant, a training-free KV cache compression algorithm released in March 2026 that compresses cache storage from 16 bits down to 3 bits with m

ACE-Step 1.5 XL Base is a 4B-parameter Diffusion Transformer (DiT) decoder in the ACE-Step 1.5 open-source music generation framework, designed to deliver higher audio quality than the earlier 2B-p...

A factual description for this entry would be: This model is a base version of Qwen 3.5 with a 4-billion parameter count, optimized using ZitGen techniques. It is likely used for supplementary or f...