Knowledge catalogue
Another PR of ours just merged into llama.cpp: the fast Metal kernels from Monday now cover all 26 weight formats the Mac GPU runs. They made speculative decoding faster on Mac for 10 formats, and thi
It is very cool to see llama.cpp on the big stage in todays Windows event! The software and hardware stacks are finally coming together. Our community has put a lot of hard work in the past years and
llama.cpp can distribute inference on heterogeneous devices through the ggml RPC backend It's an advanced setting but I think with time we'll make it more accessible to regular users. It's crazy that
The Windows launch event today has mentioned 'local AI' 41 times (so far) 🤯 Media It is very cool to see llama.cpp on the big stage in todays Windows event! The software and hardware stacks are finall
On October 6 2026, Georgi Gerganov ( @ggerganov ) announced **day‑0 support for EmbeddingGemma 2 in llama.cpp** on Hugging Face. The announcement linked to the GGUF‑formatted model weights hosted unde
decision models + local ai = ❤️ tried Laya (421M) on llama.cpp in a little apple-picking benchmark, it's almost as good as Qwen 4B at 12x the speed. Media Decision models in llama.cpp are now availabl
If you are using Qwen3.8-27B + MTP, make sure to upgrade to DFlash for extra speed: llama serve -hf ggml-org/Qwen3.8-27B-GGUF --spec-type draft-dflash --spec-draft-n-max 7 Requires the latest llama.cp
releasing my local AI slide deck covers from prefill vs decode, MoE vs dense, VRAM vs unified memory, quantization to speculative decoding, everything with llama.cpp feel free to reuse with attributio
You can now run Clef via llama.cpp! llama serve -hf ggml-org/Clef-GGUF Ollama vs llama.cpp speed test on my M5 Max MBP: The new v0.6.0 release packs a lot of good stuff: - Clef support (text + vision)
We can't thank llama.cpp and @ggerganov enough for being among the pioneers of local AI. But we sure as hell can contribute back! Upstream contributions will now become a priority in @qvac. One of our
Run GGUF models directly with transformers. This work brings ggml's Metal kernels to the transformers ecosystem, increasing compatibility and performance. More info below Millions of GGUF downloads la
Super happy to officially announce that we are a̶c̶q̶u̶i̶r̶i̶n̶g̶ joining forces with @nvidia 🔥 Here is a more personal take: AI is at an inflection point. Open source AI can become less relevant in t
Introducing Fleet: GPU benchmarking entirely in your browser. Run WebGPU compute kernels drawn from real AI workloads directly on your hardware and earn a personalized card built for your device. Rate
Georgi Gerganov showcased running a LLaMA server on an NVIDIA DGX‑Spark, serving the Qwen 3.8‑27B model in GGUF format (`ggml-org/Qwen3.8-27B-GGUF`). The command demonstrates two quantization options—
Meta released Muse Glimmer 30B: multimodal model for your Claw/Pi setups 🔥 we tested and fine-tuned the model for you, and shipped day-0 support in transformers and llama.cpp, including DFlash for 2-4
Zed editor has released a new version with integrated support for llama.cpp as a model provider, enabling users to run local language models within the editor. This update allows developers to use lla
This post acknowledges a pull request contribution (#59964) to the Zed code editor repository by user hanouticelina, as shared by Georgi Gerganov on X (formerly Twitter). The specific details of what
llama.cpp recently added DFlash support to its speculative decoding arsenal. Along with MTP, Eagle3 and various ngram-based techniques, the local model performance takes another step up. Special thank
Speculative decoding is a technique implemented in llama.cpp that speeds up inference by using a smaller, faster model to predict multiple tokens ahead, which a larger model then verifies in parallel,
In HF GGUF section of models, we are emphasizing MTP heads with its own sign 𝗠𝗧𝗣 llama.cpp adds MTP for the Qwen3.6 family This is a significant milestone for the local AI ecosystem. The performance j
Gemma 4 MTP (Multi-Token Prediction) has been officially integrated into llama.cpp, enabling developers to build faster AI experiences by using the model with this inference framework. This merge allo
Highlighting recent advances in multi-GPU and tensor parallel support in llama.cpp Over the last few months llama.cpp maintainers and engineers from NVIDIA collaborated to improve the multi-GPU perfor
Computex 2024 is expected to feature significant announcements around local AI deployment, with major industry players including NVIDIA and Microsoft actively promoting and discussing on-device AI wor
These are some of my LLM assisted contributions from the past month. Nothing amazing, but I'm slowly getting better at it. Atm, using Qwen3.6 27B exclusively. For hardware - switching between M2 Ultra
Highlighting the new WebGPU backend in llama.cpp/ggml The work to bring full-fledged WebGPU support in llama.cpp started about an year and a half ago. It has been lead by @reeselevine and team at USCS
We have an arxiv paper up describing the work in more detail here: https://arxiv.org/abs/2605.20706. Also want to call out that there is even more room for improvement, some recent updates to wllama b
What hardware actually powers open-source AI? Not benchmarks. Not vendor marketing. Real-world community usage. We’re launching @huggingface Hardware: → trending GPUs & CPUs → VRAM distribution → infe
Llama-eval is a proposed evaluation tool for llama.cpp designed to standardize and improve comparability of community-run evaluations. The tool aims to address inconsistencies in how different users b
@BereznevKi20669 @ggerganov Yes I believe the real llama.cpp revolution is yet to happen at its full scale. As computers will have more RAM and models will improve, and *if* China will continue shippi
This post documents a technical demonstration of running the large Qwen3.5-397B-A17B model across distributed hardware using llama.cpp with advanced networking protocols. The approach leverages 4-bit
llama.cpp now supports various small OCR models that can run on low-end devices. These models are small enough to run on GPU with 4GB VRAM, and some of them can even run on CPU with decent performance
sam3.cpp - Meta's SAM 3 in pure C++ with @ggerganov's ggml - Supports SAM 3.1, 3, 2.1, 2 and EdgeTAM - FP16, 4-bit quant (EdgeTAM in 15 MB) - Apple Metal GPU, CUDA, CPU - Text-prompted: 'peach' → ever