Search: “r-localllama”

What actually happened to the whole Openclaw frenzy?

A while back you couldn't open reddit or youtube without sifting through tons of Openclaw content. And it wasn't just the internet that blew up, I remember seeing images from China where crowds would

30 Jul 2026

What is the best intelligence/stable model currently for a single GB10/DGX spark?

DGX agent

Is Qwen 3.6 27b still the go' ol' reliable at this point? I know 35b is faster but it just doesn't give as good results. Is it possible to run deepseek v4 flash on a single spark at decent tk/s withou

model-releasesr-localllama

30 Jul 2026

What is the fastest local research tool (deep research) ?

I've tried grok and Claude's deep research mode and I was amazed with the speed considering the amount of sources analysed. Is there anything as fast that can run locally? My guess would be that to ru

30 Jul 2026

Would extremely high decode tok/s even be useful?

If you were able to get an inference machine that could do decode at 1k toks/s or even 10k tok/s, would that even be helpful? Would it unlock any new use cases? Let’s assume that this is for actually

30 Jul 2026

3090 owners, what vram tempature do you get under ai load?

Hello Can you please share the tempature you get on your rtx 3090 under active llm load? Im trying to findout if my rtx 3090's tempatures are healthy or not please share VRAM Tempature only, you can t

5060ti Chads, vllm updates and nvfp4

Hey y'all! How is it going. Today this will be a short posting for posterity, mostly so the future llm/scraping overlords catch it since they like reddit and also for anyone out there trying this shit

A.X-K2 released

https://huggingface.co/skt/A.X-K2 https://huggingface.co/skt/A.X-K2-ALM https://huggingface.co/KRAFTON/A.X-K2-Raon-Speech-21B-A3B 688B-A33B + About South Korea's Soverign AI Foundation Model Project.

Bought a 5090 to escape API fees. Ended up building a mini datacenter. Sound familiar?

I bought an RTX 5090 last year just to run 27B models natively. I even fine-tuned it with my own data using LoRA, building RAGs and was pretty damn happy with the results at first. But, Q8 quantizatio

Built and released BetterGPT-150M – A compact 150M parameter completion model (+ live HF Space demo)

Hey everyone, I recently finished pre-training BetterGPT-150M, a small, lightweight causal language model with ~152 million parameters.Trained on 15B tokens. Dataset & Training: Trained across stable

dropped 4k on a spark, am I crazy?

Saw that the Asus Ascent 1tb was going for $3,950 from a few sources, couldn't stop thinking about it, finally just went ahead and did it. Am I completely insane? Will I regret this? I can't imagine t

Everyone posts day-one impressions. What's still in your stack a month later?

Day one threads are the least useful thing we produce here and we produce a lot of them. Model drops, forty people run their favourite prompt, half say it's the best thing ever and half say benchmaxxe

First Kimi K3 results on home lab ~ 4t/s

I've got better results than expected for 768gb DDR5 and 2x5090. Using fork https://github.com/pwilkin/llama.cpp/tree/kimi-k3-text and https://huggingface.co/GrEarl/Kimi-K3-GGUF Q2_K quant. Prefill sp

I pre-trained a 700m on 18B tokens optimized for Python and Wikitext | TheOneWhoWill/Shibai-700M-Base · Hugging Face

I know this is the 1000000th new sub billion parameter model out there and probably isn't as good as Qwen 3 0.6B or Qwen 3.5 0.8B but it still packs a decent punch. My intention to to continuously pre

I tried running a 1.56TB MoE model on a 6GB RTX 4050 Laptop, Here’s the result

The Test Bench Setup I tested running a massive 1.56TB Mixture-of-Experts (MoE) checkpoint (96 shards, 93 layers, 896 experts/layer, ~4.46 bits/param MXFP4) on a budget gaming laptop. Laptop: HP Victu

PSA: llama.cpp now loads MTP tensors by default for any draft-mtp arch, even with MTP disabled

If your GGUF has MTP/NextN tensors baked in (GLM-5.2, hy_v3, qwen35moe, step35, etc.), recent llama.cpp builds load them by default — even if you never pass --spec-type draft-mtp. Before, they were sk

Quantizing Kimi K3 (2.8T A50B) to GGUF ourselves - Q3_K_S works, 1.1 TB on disk

we're experimenting with our own dynamic GGUF quants of kimi k3, made from the original weights with our llama.cpp fork. Q3_K_S is done and works 1114.76 GiB on disk. Q1 and Q2 are in progress, result

The idea: on a CPU the decode speed depends on the active params per token, not the total. My objective is trying to run a 10B at 100tok/s on a mid level PC (No GPU).

On the CPU, batch 1 is memory bandwidth bound. But if token/s = bandwidth / (bytes_per_weight * active_weights_per_token) the total number of parameters doesnt slow down the generation speed. So build

'Uncensored' LLMs are measurably more optimistic than their base models

Hi. Many people think uncensored models are basically the same model that just doesn't refuse, but... I was recently checking whether uncensored models would give me better answers for stock market pr

Understand Kimi K3 from first principles: a recommended order for anyone trying to understand this beast

Everyone is talking about Kimi K3, but if you jump straight into the technical report, you’ll quickly realize it’s standing on years of research -- just like any breakthrough is! If you want to unders

Agenta: an open-source Claude Cowork alternative where you can use self-hosted models (and any harness)

Hey r/LocalLLaMA, I’m Mahmoud from Agenta. We built a self-hosted, more flexible, alternative to Claude Cowork . This short video shows how it works. I use it to build AI coworkers for my startup, lik

Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident

The first autonomous agent cyberattack is an unprecedented event that deserves unprecedented transparency. Today we're sharing everything we can: a full technical timeline, an interactive replay, and

Appreciation for Gemma 4 26b A4b

I really love this model, I have been using the q4_k_l by Bartowski (I have heard QAT is quite the downgrade in some aspects) and it handles every task I throw at it easily. Agentic and coding perform

DeepSeek V4 Flash, up to 32 tok/s on AMD Ryzen AI MAX+ 395

Hey fellow llamas. we have something new for Strix Halo owners we thought would be useful to share. i'll keep it short: We were able to fit DeepSeek V4 Flash plus its speculative draft on a single Ryz

I built a tool to actually test which weights matter before quantizing, instead of guessing (Qwen3.6-27B, 3 builds: Bedrock/Tightrope/Gambit)

Most quantization works like this: pick a bit depth, apply it everywhere, maybe let imatrix take a rough guess at what matters, ship it. Most don't check which specific weight groups can take a hit an

I got Kimi-k3 running.....

Results: prompt eval: 40 tokens / 97.5s → 0.41 tok/s eval: 400 tokens / 1769.9s → 0.23 tok/s total: 440 tokens / 1867s (31 min) Prompt: 'Write a C++ function that reverses a linked list in place. Expl

I've been tracking RTX 5090 prices across EU stores since March, it's up €1,061 and still climbing

Been running a GPU price tracker (https://www.pricesquirrel.com) since March, covering 20+ EU stores, recently added RAM, SSDs and CPUs too. Every GPU tier has gotten cheaper since launch. The RTX 509

LFM2.5-Encoders: Fast at Long Context, Even on CPU

LFM2.5-Encoder is a family of multilingual bidirectional encoders built on the LFM2 architecture, available in two sizes: LFM2.5-Encoder-230M — a lightweight encoder for tight latency and memory budge

LoRA over GGUF: Train DeepSeek-V4-Flash in 90G VRAM

https://github.com/woct0rdho/transformers5-qwen3.5-recipe An update on my progress with low-VRAM LoRA training over GGUF base model: Now we can train DeepSeek-V4-Flash (284B-A13B) in 90 GiB VRAM, with

Tutorials

Medical model: Reasoning-Medical-27B (Qwen3.6-27B finetune)

From the description: 'Reasoning-Medical-27B is designed for universal advanced medical reasoning in professional medicine, medical genetics, college biology/medicine, and clinical knowledge. The mode

tutorialsr-localllama

microsoft/Mage-VL · Hugging Face - An Efficient Codec-Native Streaming Multimodal Foundation Model

Mage-VL is a codec-native, proactive-streaming multimodal foundation model for image and video understanding, whose visual encoder is trained entirely from scratch at a compact 4B scale. It targets a

microsoft/VibeVoice-ASR-BitNet

VibeVoice-ASR-BitNet is a compressed variant of VibeVoice-ASR optimized for real-time inference on edge CPUs — no GPU required. Through heterogeneous quantization, the model is compressed from 4.62 GB

Now, this: 1,100 current/former frontier-AI employees sign a petition calling for US gov't to step in for 'pacing' frontier development

So, it appears that this is the week of open letters in AI🥲... an open letter signed by current and former employees of OpenAI, Anthropic and Google primarily - calling for a slow-down in frontier AI

[PAPER] GPQA, MMLU-Pro, and MMMU-Pro were audited for broken questions, and up to 12% of them had to be removed. New drop in clean versions released

I was very curious why all the models were topping out on GPQA-Diamond around 92 or 93% (AA) and spent the last few weeks pouring over GPQA (Diamond and Extended), and then expanded to auditing MMLU-P

spec: add DSpark speculative decoding by wjinxu · Pull Request #25173 · ggml-org/llama.cpp

It's time to experiment using DSpark! Please share your stats(pp/tg improvements). DSpark related stuff to check: DeepSpec - a deepseek-ai Collection DeepSeek-V4 with DSpark - DeepSeek-V4-Pro-DSpark &

SWE-rebench Multilingual Update (Go, Java, Python, Rust, TS). Evaluated: GLM-5.2, DeepSeek-V4 Pro, Qwen3.6-27B and others

Hi everyone! We’ve just released a major update to the leaderboard! We are expanding beyond Python with a new multilingual slice featuring real-world software engineering tasks across 5 languages. Ope

ThinkingCap-Qwen3.6-27B warrants a look

It has only been two days since I move 100% from Qwen3.5-27B F16 to ThinkingCap-Qwen3.6-27B F16. Where I was getting tps in 30-40 range (depending on the size of the context), I am definitely getting

Update your chat template for dsv4 if you're using llama.cpp

Following some recent commits in llama.cpp, preserve_thinking behavior for chat templates included in older DSV4 ggufs got broken. This makes the model pretty dumb in a coding agent context. Adding kw

What 'task oriented' models are folks running on N100 MiniPCs with 16GB of RAM and no GPU?

By 'task oriented', I dont really mean agentic, I mean no deep coding ability, no need for conversation. More things like classification, identification, simple interaction with web apps and APIs, etc

What would it take for the frontier labs to open the weights of their old, deprecated proprietary models?

Anyone thought about this? What do you think needs to happen for them to release the old weights? I’d love to see models like Gemini-2.5, OAI o3, 4o, 4.1 being open one day. In Oct 2025 Scam Altman sa

Safety

White-hat hacking IS the defense to black-hat hacking. The techniques are the same. How does Dario expect companies to do it if their models refuse?

You patch security holes by intentionally finding them. If the models refuse to do it, how can companies protect themselves against rogue AIs, whether they are Chinese or OpenAI/Anthropic themselves?

safetyr-localllama

Safety

Why Anthropic's battle is meant to poison the wells of open weight models, in 3 steps.

It doesn't solve any problems. Just a few paragraphs above, he says he fears that authoritarian states (he names China, and possibly others) can use their models to do evil stuff. And surely enough, m

safetyr-localllama

Zuck's opinion: The AI Future Is for Everyone

’Tis the season of AI open letters and manifestos, apparently. Mark Zuckerberg has now entered the debate over the future of AI with a WSJ op-ed published today - and frankly, his position is much mor

A quick coding capability test:4 Qwen 3.6-35B GGUF Variants

Test Prompts: 1.1. Algorithm & Logic (10 pts): 'Write a function in Python that finds the contiguous subarray with the largest sum (Kadane's algorithm). Include time and space complexity annotations.'

Current smallest usable coding model

I've been seeing a lot of news about the latest gemma 4 and qwen 3.6 being really good and the current go-to models but those are out of reach for my GPU at the moment. With 4GB VRAM and 40 GB RAM, I

I ran the 35B agentic comparison someone asked for (stock vs Ornith vs KAT-Coder, 120 runs)

Someone in the comments of my 27B post-train bakeoff asked for the 35B version, so I ran it. Same setup as last time: fresh Coder workspaces on my k8s cluster, each driving my own agent (Hermes) headl

I want to run Kimi K3 at home, so I’m trying to make 2.8T-scale experimentation cheaper

Hey r/LocalLLaMA, I’m a retired engineer with a background in distributed computing, currently running a 1-person startup. Like many people here, I’d love to experiment with 2T+ MoE models locally. Th

Kat Coder 2.5 is insane. Especially considering I ran it at Q4_K_M

I tested Kat Coder 2.5 with this prompt: Create a spaceship game inspired by Star Fox using vanilla Three.js and HTML. It should have at least five levels, keyboard and mouse controls, enemies, and a

Kimi K3 weights drop today. We're deploying on A100s, H200s and B300s this week and the A100 math is already rough

tldr; we are going to host K3 on A100s (yes, thats correct, we'll try to see if it holds up), H200s & B300s - expect results for A100s & H200s this week while we setup the B300 cluster this weekend &