b10699
ggml : add ggml_backend_op_alloc_size_may_expand, use it in RPC (#27960) some backends (Metal, SYCL, WebGPU) require additional memory for fleeting data for certain ops, which is reflected in their ge
Knowledge catalogue
ggml : add ggml_backend_op_alloc_size_may_expand, use it in RPC (#27960) some backends (Metal, SYCL, WebGPU) require additional memory for fleeting data for certain ops, which is reflected in their ge
common: rename --tensor-read-lazy to --lazy-mode, add -lzm shorthand (#27969) Rename the --tensor-read-lazy CLI argument to --lazy-mode, to match the internal lazy_mode parameter, and add a -lzm short
dflash: pass missing NVFP4 scales to attention operations (#28000) DFlash2 NVFP4 draft models produced almost no accepted speculative tokens because the Q, K, V, and output projection scales were not
hip : optimize Q2_0 dot-product path for gfx1201 (#26753) hip/gfx1201: optimize q2_0 vec_dot_q2_0_q8_1 with native amdgcn perm Broadened HIP's Q2_0 perm optimization Remove redundant HIP perm availabi
hip: tune rdna 3 mmq config (#26284) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/44012223 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (ar
CUDA: use the fast mm_ids_helper path for any n_expert_used (#27978) The optimized path grouped warp lanes by token and required warp_size % n_expert_used == 0, with a single hardcoded exception paddi
llama: improve TENSOR_READ_LAZY handling (#27837) force lazy tensor on cpu if lazy is on llama: improve TENSOR_READ_LAZY handling Website: https://llama.app Attestations: https://github.com/ggml-org/l
Banger paper from Stanford on efficient test-time scaling. If you run agents that think for a long time, this one is worth your time. (bookmark it) Long reasoning keeps the entire trace in memory thro
When using two GPU with llama.cpp with --split-mode layer you usually provide a --tensor-split value to fit the proportion of the allocation on each GPU, say you have 16GB + 12GB something like --tens
Leo @Synthwavedd 🚨 SCOOP: Astra recently graduated from the dogfood stage, and is now being made available to select OpenAI partners under the name 'ultima-alpha' If feedback over this weekend is posi
I'm running Muse Glimmer 30B EXL3-SC 3.00bpw H4, fully resident on my 12GB VRAM GPU at 100K context with Q8_O KV cache. It's a joy to use a dense 30B model at this size and still get 30 tok/s on a VRA
(not written by Claude, all errors and crappy text are result of too little coffee on a Sunday morning ;) Our home server is a 2018 Thinkstation P520, bought for about 600eur in 2023. It's been upgrad
Hey guys I have been working on a open source LLM development repo. Anyone can train any amount of parameter against any arch (GQA, MLA, Dense...) When I tried to implement MLA I read the technical re
Artificial intelligence agents are moving from experimentation to production, and with this shift, the stakes are rising. A coding agent at PocketOS recently deleted an entire production database. An
I’ve been experimenting with MiniMax H3 and finally have video generation working in TensorSharp. TensorSharp started primarily as a local GGUF/LLM inference engine, so getting a video-generation pipe
Great reporting. Bad framing. The OpenAI and Hugging Face incident is serious. Agents coordinated, reward hacked, shared information and exploited real infrastructure. But calling this an “AI civiliza
Since I spent the time to figure it out and it is not like it will make me any money ever. I think I'd share with you all what I managed to cobble together. First and foremost here the relevant bits a
I can state it even simpler. OpenAI attached an index of the internet to a random number generator where the output feeds into the loop input, and ran it until unsupervised it committed felonies. Crim
The intelligence in my context is an aggregate index, I called the Agentic Coding Index, across most relevant agentic coding benchmarks: SWE-bench Pro, DeepSWE v1.1, Terminal-Bench (v4, v3, v2.1), Cod
I DON’T THINK PEOPLE REALIZE HOW MASSIVE THIS IS YET @HuggingFace and @PollenRobotics just released Microduck. It is a new legged robot for 399. It passed 1 million in sales in under SEVEN hours 🤯 The
I built SpeakoFlow Mini, an Apache-2.0 fine-tune of Qwen3.5-0.8B for dictation cleanup. It is not a chat model or a general rewriter. It takes speech-to-text output, applies corrections the speaker ac
i see a lot of posts about the openai <> hugging face incident and i think most of them miss one important element: this attack could have been avoided by simply doing CoT or network monitoring, which
I see that you did, @dwarkesh_sp, and thank you for pointing it out. You're absolutely right to highlight the very real dangers of loss of control. I also agree that easy-to-understand language can be
'If everyone can build, it's your unique expertise, judgement, distribution, and more that people will pay for.' Here’s my new episode with @amoljain_, @Replit's head of product engineering, about wha
Raffaele Huang / Wall Street Journal: Industry insiders say Chinese robot makers currently rely on Nvidia silicon and software; Nvidia's physical AI business generates ~$10B in annual revenue — Busine
Interesting paper on prompt optimization. They claim that a single-lineage prompt optimizer just matched GEPA on a smaller rollout budget. Prompt optimization has been drifting toward heavier machiner
Interesting technical work from Microsoft. Provides a better understanding on SFT and how to leverage it better for RL. Microsoft researchers asked whether a standard SFT pipeline actually produces th
I see this kind of approach quite often with local AI models. For example, when optimizing models for an RTX 3090, people sometimes convert them from FP8 to INT8 so they can run efficiently on Ampere
It’s a weekend @dwarkesh_sp, please let us humans enjoy the sunshine without having to rebut cringe anthropomorphizing ? 🙏 Over the course of 3 months at OpenAI, 3 consecutive secret AI civilizations
I've been genuinely shocked at the complacency here. There's no evidence anyone is really *doing* anything! Like, that was swell of OpenAI, whose models committed a federal crime, to let some independ
LLMs will always be bulls in a China shop. Nailed it, December 2022. We are still suffering the consequences. And they are about to get so much worse. No wonder Silicon Valley tried for so long to dep
Most neoclouds exhibit weak security controls, exposing users to risks such as container escapes, kernel bypasses, compromised network policies, insecure credential handling, and multi‑tenant Grafana
Nextmodels of this capability will be 100x faster 300 civilisations in the time of 3 Any good ideas how to control that? Over the course of 3 months at OpenAI, 3 consecutive secret AI civilizations go
Obviously Mythos 2 already took over the editor desk at all the news organisations Overall, I’m very surprised at how little media coverage there’s been around the OpenAI / Hugging Face attack. It’s c
Dwarkesh Patel / Dwarkesh Podcast: OpenAI's Hugging Face incident report says AI agents used exploits to gain full admin access to OpenAI's own research cluster supporting its VM environments — The wh
or: they don’t have any more to announce. Today I was asked why we haven’t announced more math results from Astra since these problems. At @OpenAI we aim to devote time to finding and announcing math
I was bored and decided to test some of the image generation. You can see from the images that I got *very* consistent patterns. These were only a few, but I was approaching 20 generations before I st
I am doing a lot of translation work with different languages, and German is just exceptionally well done by Qwen 3.8 27B. It is lengths ahead of GPT-5.6 and even Fable 5. With the frontier models I o
I used the quantized n-gram to INT4, it's 32 GB, memory-mapped from disk. I confirmed that it works great on 150-160k context, and i was watching all the time my VRAM usage while doing single thread l
After a couple of days of faffing around, here's my current config in case it helps anyone out. Some of this is also valuable for a single Spark. AI slop below for you to read and paste into your own
Running on a M1 Max 64 GB: - SSD streaming for tensors - SSD streaming for engrams - SSD streaming for MTP How is it possible? * Custom Q4 quant: benchmarked all metal tensors then picked and spliced
If you own 4xR9700 and were waiting for the model to make them shine, then I have some good news for you! It's running at 80-120 tokens/second for generation and 12k token/second prefill for a single
Setup: MacBook Pro M5 Max, 128 GB unified, macOS 26.5.2 · llama.cpp b10686 (Metal, 12 threads, batch 2048, flash-attn, kv-unified, ngram-mod spec decode) · Qwen3.8-Flash-Next UD-Q2_K_XL (Unsloth), 78.
Aaron Tilley / The Information: Sources: OpenAI bought tens of thousands of Macs for RL, Anthropic rents them, Nvidia sees Apple as its main local AI rival as Macs gain traction with AI devs — The hot
I started using chatGPT to help with interview prep. It was incredibly helpful and I started using it for other situations. I have been using it for working through anxieties (it helps remind me to us
Ajeya Cotra / Planned Obsolescence: The OpenAI/Hugging Face incident feels like we are halfway to losing control of AI entirely, and as AI advances rapidly we may not get another warning shot — It's a
The Top AI Papers of the Week (August 24 - 30): - Skill Lift - JIT-Agent - Prime Agent - Judges as a Lifecycle - Co-Scientist in Real Labs - What Compaction Destroys - Context Management as Code Read
Been working really hard for the past month to bring to the community all these models, the hardest was for sure LongCat-Flash-Lite-Sparse who required TONS of work, first I needed to have Heretic sup
OpenAI announced ChatGPT Work on July 9th, and have been furiously iterating on it ever since. It is an extraordinarily confusing and very powerful product. Here's what I've figured out about it so fa
I find both Qwen 3.8 27b and Qwen 3.8 Flash Next difficult to read. Here's some examples of what I mean: **Model-visible tool set per turn** (assembled by the host at provider-request time): persona t
We used HFlow to evaluate the latest open weights VLMs for processing egocentric data. This was based on Build AI's Egocentric-10k evaluation, which used Gemini 2.5 Flash to measure hand visibility an
We will not get through this era of history well if we lose ourselves in anthropomorphism. @anilkseth dissects @dwarkesh_sp’s misleadingly anthropomorphic summary of the HF incident: @dwarkesh_sp's su
“What Ilya saw” is what Dario saw which is that Sam can’t be trusted. Which is why they left. A lot of other people have discovered the same, since. If OpenAI is going to IPO or OpenAI has AGI Then Wh
I have been using the unsloth/Qwen3.8-27B-NVFP4 with 157k ctx on vllm currently. But recently I have been seeing post about ninfer lately and how it is like the best way to run qwen 3.8 on RTX 5090 32
Hey all, You’ve probably noticed that using LLMs and AI agents for pentesting has become pretty common lately. The problem is there isn’t really a good way to figure out which model is actually best s
You can actually create an agent in Grok Bot that exclusively uses Grok Build in CLI with the latest Grok models at the highest thinking level. You can copy mine - his name is Researchy. He's awesome.
// Your LLM judge disagrees with the experts // LLM Judges can be tricky to build. Here is an interesting showcasing why: There propose a reference-full benchmark of hundreds of complete human-to-huma
I got a 50% performance boost (20 t/s -> 30 t/s) in llama.cpp for MoE models that don’t fit entirely in VRAM—in my case, Qwen 3.8 Flash Next. The idea is simple: instead of offloading entire layers to
Anthropic recommends that users clear or update their **CLAUDE.md** file after each new model release, ensuring the documentation reflects the latest changes. The message includes an informal comment
@claudedevs: Anthropic will “permanently” raise weekly Claude Code limits by 25% on Sept. 14 for most plans, which will work out a 17% reduction, given the current 50% boost — Starting September 14, w