Search: “georgi-gerganov--x”

llama.cpp can distribute inference on heterogeneous devices through the ggml RPC backend It's an advanced setting but I think with time we'll make it more accessible to regular users. It's crazy that

The Windows launch event today has mentioned 'local AI' 41 times (so far) 🤯 Media It is very cool to see llama.cpp on the big stage in todays Windows event! The software and hardware stacks are finall

On October 6 2026, Georgi Gerganov ( @ggerganov ) announced **day‑0 support for EmbeddingGemma 2 in llama.cpp** on Hugging Face. The announcement linked to the GGUF‑formatted model weights hosted unde

decision models + local ai = ❤️ tried Laya (421M) on llama.cpp in a little apple-picking benchmark, it's almost as good as Qwen 4B at 12x the speed. Media Decision models in llama.cpp are now availabl

If you are using Qwen3.8-27B + MTP, make sure to upgrade to DFlash for extra speed: llama serve -hf ggml-org/Qwen3.8-27B-GGUF --spec-type draft-dflash --spec-draft-n-max 7 Requires the latest llama.cp

releasing my local AI slide deck covers from prefill vs decode, MoE vs dense, VRAM vs unified memory, quantization to speculative decoding, everything with llama.cpp feel free to reuse with attributio

You can now run Clef via llama.cpp! llama serve -hf ggml-org/Clef-GGUF Ollama vs llama.cpp speed test on my M5 Max MBP: The new v0.6.0 release packs a lot of good stuff: - Clef support (text + vision)

We can't thank llama.cpp and @ggerganov enough for being among the pioneers of local AI. But we sure as hell can contribute back! Upstream contributions will now become a priority in @qvac. One of our

Run GGUF models directly with transformers. This work brings ggml's Metal kernels to the transformers ecosystem, increasing compatibility and performance. More info below Millions of GGUF downloads la

Super happy to officially announce that we are a̶c̶q̶u̶i̶r̶i̶n̶g̶ joining forces with @nvidia 🔥 Here is a more personal take: AI is at an inflection point. Open source AI can become less relevant in t

Introducing Fleet: GPU benchmarking entirely in your browser. Run WebGPU compute kernels drawn from real AI workloads directly on your hardware and earn a personalized card built for your device. Rate

Georgi Gerganov showcased running a LLaMA server on an NVIDIA DGX‑Spark, serving the Qwen 3.8‑27B model in GGUF format (`ggml-org/Qwen3.8-27B-GGUF`). The command demonstrates two quantization options—

Meta released Muse Glimmer 30B: multimodal model for your Claw/Pi setups 🔥 we tested and fine-tuned the model for you, and shipped day-0 support in transformers and llama.cpp, including DFlash for 2-4

Zed editor has released a new version with integrated support for llama.cpp as a model provider, enabling users to run local language models within the editor. This update allows developers to use lla

This post acknowledges a pull request contribution (#59964) to the Zed code editor repository by user hanouticelina, as shared by Georgi Gerganov on X (formerly Twitter). The specific details of what

llama.cpp recently added DFlash support to its speculative decoding arsenal. Along with MTP, Eagle3 and various ngram-based techniques, the local model performance takes another step up. Special thank

Speculative decoding is a technique implemented in llama.cpp that speeds up inference by using a smaller, faster model to predict multiple tokens ahead, which a larger model then verifies in parallel,

In HF GGUF section of models, we are emphasizing MTP heads with its own sign 𝗠𝗧𝗣 llama.cpp adds MTP for the Qwen3.6 family This is a significant milestone for the local AI ecosystem. The performance j

Gemma 4 MTP (Multi-Token Prediction) has been officially integrated into llama.cpp, enabling developers to build faster AI experiences by using the model with this inference framework. This merge allo

Highlighting recent advances in multi-GPU and tensor parallel support in llama.cpp Over the last few months llama.cpp maintainers and engineers from NVIDIA collaborated to improve the multi-GPU perfor

Computex 2024 is expected to feature significant announcements around local AI deployment, with major industry players including NVIDIA and Microsoft actively promoting and discussing on-device AI wor

These are some of my LLM assisted contributions from the past month. Nothing amazing, but I'm slowly getting better at it. Atm, using Qwen3.6 27B exclusively. For hardware - switching between M2 Ultra

Highlighting the new WebGPU backend in llama.cpp/ggml The work to bring full-fledged WebGPU support in llama.cpp started about an year and a half ago. It has been lead by @reeselevine and team at USCS

We have an arxiv paper up describing the work in more detail here: https://arxiv.org/abs/2605.20706. Also want to call out that there is even more room for improvement, some recent updates to wllama b

What hardware actually powers open-source AI? Not benchmarks. Not vendor marketing. Real-world community usage. We’re launching @huggingface Hardware: → trending GPUs & CPUs → VRAM distribution → infe

Llama-eval is a proposed evaluation tool for llama.cpp designed to standardize and improve comparability of community-run evaluations. The tool aims to address inconsistencies in how different users b

@BereznevKi20669 @ggerganov Yes I believe the real llama.cpp revolution is yet to happen at its full scale. As computers will have more RAM and models will improve, and *if* China will continue shippi

This post documents a technical demonstration of running the large Qwen3.5-397B-A17B model across distributed hardware using llama.cpp with advanced networking protocols. The approach leverages 4-bit

llama.cpp now supports various small OCR models that can run on low-end devices. These models are small enough to run on GPU with 4GB VRAM, and some of them can even run on CPU with decent performance

sam3.cpp - Meta's SAM 3 in pure C++ with @ggerganov's ggml - Supports SAM 3.1, 3, 2.1, 2 and EdgeTAM - FP16, 4-bit quant (EdgeTAM in 15 MB) - Apple Metal GPU, CUDA, CPU - Text-prompted: 'peach' → ever