Knowledge catalogue
I am planning to buy hardware to run local models. I'm not sure what kind of hardware is best and what will I be able to run. I'm torn between getting a used M1 Max 64gb Macbook for €1500, M3 Max 128g
I wanted to test what Qwen 3.8 27B running locally on a single 5090 could do in Row-Bot. I gave it just one prompt: 'I want you to create an animation showing what the current local model can do. Test
Follow up post of https://www.reddit.com/r/LocalLLaMA/s/nGMyKswrch. Thanks everyone who replied. I didn't change the specs. Might be loosing some of the memory bandwidth but will scale in future if I
What taste in humor do LLMs have? Do LLMs like certain types of jokes? Do they have strong opinions? Our report: LLMs have some overlap in humor taste, about 3/4 agreement but it ranges from 54.9% to
With the recent Nvidia Huggingface acquisition and frontier AI labs screaming about safety and putting guardrails everywhere, I think it's important that we have local models that aren't affected by a
I've been working on something for this community (and other self-hosted communities) for almost 5 months now. Personally, I would grade it as probably the closest open source equivalent of Manus and
https://reddit.com/link/1wi2zuc/video/rs1a4sx92xph1/player Hey everyone, I’ve been experimenting with local LLMs on Intel hardware, and I wanted a hyper-optimized, standalone CLI tool that didn't rely
I'm pretty sure it can be done with any model based on qwen4exp, which Qwen's next local models will be based on. You can use a quant that barely fits in VRAM and still run at the model's maximum cont
I implemented expert lookahead, a trick that gets a 10%+ performance improvement on MoE models running on low-memory macs (Qwen 3.8 flash in this case) while using expert-offloading/ssd-streaming (usi
So I would like to run qwen 3.8 27b locally for my ai agents, maybe even in parallel with other small LLMs (such as qwen3.5 9b, oss 20b etc..). What is the best hardware to do this? Not a video card b
Hey r/LocalLLaMA, We’ve released our full ShapeLearn GGUFs for Qwen 3.8 27B. Blog / Download models TL;DR 3.84 bpw (GPU-5) reaches 99.63% of BF16’s aggregate score of 8 benchmarks, being the most accu
example: https://huggingface.co/audnai/penclaw-GLM-5.3-abliterated-for-offensive-cyber. edit: good context for why this was done here https://www.reddit.com/r/LocalLLaMA/comments/1whbvpx/comment/pa16g
Disclaimer: no AI was used whatsoever to write this post Cautionary tale about chasing cheap tokens. exposé: https://kendell.dev/blog/crofaifalse/ reaction by nahcrof, announcing the shutdown of the s
EDIT: Sorry for the unclear title. This model is UkisAI's Swift-Qwen3.8-27B, not a new version of BottleCap AI's 3.6-ThinkingCap. All credit goes to UkisAI for making great fine-tune, and I made this
I liked DeepSeek V4.1 Flash as an agent model but at 16 t/s on ds4 it was painful to sit through a real turn. It seemed some redditors and m3 ultra owners appreciated my glm 53 flash optimizations, so
One thing I love about both Qwen 3.8 27B and Flash Next is the effort it puts in to make a good product. It is the opposite of a lazy model and 'good enough' is never what aims to accomplish. It goes
I have tried using qwen 3.6 and 3.8 q5 64k context and I'm just not getting the experience everyone here seems to get. I'm using open code and it will loop, forget what it's doing, and mess up the ui.
After two days of fighting with benchmarking infrastructure, I finally benchmarked IFM/K2-Horizon-7B on 16GB VRAM. TL/DR: it works, but far behind Qwen-3.8-27B. Setup I compared 3 models - Qwen3.8-27B
Three weeks back , i posted SHADOW-250M here. It got 360 upvotes, 293 on r/LocalLLaMA and 94 GitHub stars. Thank you. That model was 60 MB, ran around 400 tok/s on CPU and could retrieve records from
I have a custom fork of llama.cpp designed around the ampere architecture specifically (though many of the upgrades also translate to faster performance of blackwell + lovelace). The recommended confi
For years, the JVM has watched the AI revolution from the bench. Every model, AI framework, every breakthrough, built with/for Python. jinfer is an inference engine built for the JVM from first princi
I am a beginner, Took a while to get started, get everything right. This setup is native not container. Still not sure if I did this right, or if I can tune this more. Environment=HF_HUB_OFFLINE=1 Env
As you know, Qwen3.8 27B has built-in support for embeddings. It gives pretty decent results in llama.cpp with the following parameters: --embedding --pooling mean The problem is that enabling embeddi
Hey everyone! We’re launching Voice Acting Arena with LAION x TTS Arena, and we’d love your help trying it out 🙂 You’ll hear two anonymous AI performances of the same scene and script, then compare th
Two months ago I announced I had found a new dynamic quant method called Voodoo Quant which was SOTA for the most aggressive quant levels on some smaller Qwen3.5 GGUF models. I kept the methodology pr
Personally, I think the whole 'AI (LLMs) is going to take over' is just BS marketing. They've been pushing this doom-and-gloom since 2019, and we all know the models back then were nowhere near as cap
I reallise this is not exactly a locally hostable model, at least definitly not on my hardware. But I figured some of you might have some insight on this. I run an agentic coding benchmark. The model
I had all the data saved from AA's v4.1 index, so when they upgraded it in the wake of Astra's release, I could actually generate a before/after comparison. All intelligence and price per task are sam
I know I know, it's great, we know. I've been working on tweaking inference engines for a week now and it's been one shotting most of my vague prompts without any issues. It will even write tests and
So we’re seeing how Anthropic and OpenAI are gunning for regulations. I think most of us realize that this is a ploy for them to achieve regulatory capture, thus securing their moat and kicking out op
Edit: Spelled Baseten not Base-10 Full episode of this available at https://www.youtube.com/watch?v=PrSf7IOYu-I It's interesting to see how Dwarkesh has had to come around to the evidence that we are
Do you have something you'd like to share with the r/LocalLLaMA community. This is the place for it! Recommendation on presentation: Please share plain english description of what your project does an
Note - This is translated from the actual blog link right at the bottom. A few days ago, DeepSeek v4.1 was released. It raised the ability of small models to a new level. AI is improving much faster t
https://artificialanalysis.ai/articles/artificial-analysis-intelligence-index-v4-3 AA shipped a new benchmark last week as part of the Intelligence Index v4.3 update — a brand-new private eval that re
Suppose I create two machine learning models suppose tree and neural network for a task let's suppose regression problem, now suppose I am sending both of this paper to two different journals, now the
From initial testing it seems pretty solid so far. Asked it to compile the latest llama.cpp for CUDA and its doing well so far. If this thing holds up to its score then its SHOCKINGLY good for its siz
Hi, For single-GPU training, I’m using Hugging Face SFTTrainer with auto_find_batch_size=True, which automatically reduces the batch size after a CUDA OOM until it finds a batch size that works. I wou
I'm like you guys and am constantly experimenting with new models, seeing what they're all good at, how I can make use of them for certain projects and goals. I've been using Qwen 3.8 27b for minor co
My prompt is: Epic high-fantasy concept art, ultra-detailed. A road crossing golden plains leading toward a colossal dwarven mountain hall entrance recessed into themouth of a giant cave. The whole en
Model: https://huggingface.co/internlm/Intern-S2-397B Collection: https://huggingface.co/collections/internlm/intern-s2 From Intern Large Models on 𝕏: https://x.com/intern_lm/status/209942518458737097
I'm currently using Qwen3.5 4B as the brain of my local AI assistant because my hardware is relatively limited. One thing I really like about it is the speed. On my system, I'm getting around 40–50 to
The full K2 Horizon lineup is out on Artificial Analysis. The AA intelligence vs. parameters plots show that - 0.9B and 375B are bad - 3.7B and 7B are SOTA - 36B A4B is SOTA for hardware with poor mem
Hi everyone, Inspired by tools like Chessvision.ai, I wanted to take a different architectural approach and build a browser extension (ChessInsights AI) that performs chessboard detection and piece re
I have created a VAE model using PyTorch on White Wine dataset. Basically, the main goal is to discover a brand-new white wine recipe. It puts all the wines into a latent space, finds the best part wh
I'm starting a PhD where I will be doing Graph ML (somewhere along the lines of graph signal processing/ graph deep learning.) My eventual goal is research scientist at big tech, or whichever company
Pushed out this new update, hopefully decreases the instances of crashes. I torture tested this one for ~12 hours after my fixes and found no instability. Q4 K XL needs more fine tuning, which I will
Repo here- https://github.com/Adudeguyman/ComfyUI-Fantastic-MiniMaxH3-PromptBuilder Alrighty, back again with an update to my (manual, no LLM connected) Fantastic MiniMax H3 Prompt Builder. Yeah it's
A new paper (I'm not a coauthor BTW -- I just found it interesting) argues, basically, that RSI1 is not on the horizon2, because current (at the time the study was done) agents cannot do open-ended ML
Hello Reddit. Posting this for fun. I thought it was a lonely and silly journey to set up Qwen 3.8 Next on a system that doesn't really run it properly—it was a challenge that might help the community
Now that the dust has settled a bit - here's a write-up on running Qwen3.8-Flash-Next (125B-A6B MoE + 51B n-gram table) on relatively middle-tier hardware (RTX 4070 12GB + 64GB DDR5-5600 + Gen4 NVMe o
I know Sage and Triton are nothing new but if you're a beginner or doing a fresh re-install like me, these resources helped me get faster gens on the 4-step turbo workflow I use (will post a link to t
Hi everybody, we post-trained Qwen 3.8 27B to be more efficient by figuring out which tokens were linked to overthinking and penalizing them without 'attacking' the reasoning length directly then fixe
I am considering dumping my ChatGPT Plus subscription and go full local, but to do so I would first need to reach a decent result for quality (and reasonable speed). My 4090 fried itself out of nowher
I finished my home inference server. First I tried Lenovo p620 workstation and while it’s a good value overall it pissed me off with a ton of proprietary Lenovo shit to deal with and I return it in th
Not sure if anyone out there is working on this, don't see any on huggingface to try out. It might be too new at the moment, but it would be cool to see the difference between a tiny model trained wit
The first generation of our 150M model has just been released Its performance is similar to that of GPT2-Small The benchmarks: PIQA: 62.24% Hellaswag: 32.20% Arc-Easy: 44.91% Arc-Challenge: 25.00% Ari
A few people here mentioned interest in a way to test their custom Pi setups, so I figured I’d drop this here: RoastMyHarness The basic idea is a small engine that sets up an environment to run DeepSW
Don't upvote, just looking for CPU ram options for running qwen flash next, and maybe larger models (so maybe between 128 and 256gb total ram) with an existing cmp 170hx. Use case is agentic coding so
VLLM Benchmark: Prefill, Prompt processing - avg, 871.93 tok/s (3 hours constant running xhigh) - 10K prompt, 1000.26 tok/s (16 runs) - 90K prompt, 743,59 tok/s (16 runs) Decode, tok gen - avg, 38.39
I gave DS V4.1 Flash an HLE problem with a bash tool + 2 hours. Hour 1: it wrote three MILP solvers. (225,200) Hour 2: it downloaded the HLE dataset from Hugging Face, found the question, read the ans