b10693
hexagon: support for device discovery and create sessions on demand (#27785) hex-devices: add support for lazy session allocation and cleanup dev interfaces Co-authored-by: Marco Colombo mcolombo@qti.
Knowledge catalogue
hexagon: support for device discovery and create sessions on demand (#27785) hex-devices: add support for lazy session allocation and cleanup dev interfaces Co-authored-by: Marco Colombo mcolombo@qti.
vendor : update cpp-httplib to 0.54.0 (#27919) vendor : update cpp-httplib to 0.54.0 vendor : update cpp-httplib to 0.54.0 and 0.54.1 Website: https://llama.app Attestations: https://github.com/ggml-o
ggml : add ggml_backend_op_alloc_size_may_expand, use it in RPC (#27960) some backends (Metal, SYCL, WebGPU) require additional memory for fleeting data for certain ops, which is reflected in their ge
common: rename --tensor-read-lazy to --lazy-mode, add -lzm shorthand (#27969) Rename the --tensor-read-lazy CLI argument to --lazy-mode, to match the internal lazy_mode parameter, and add a -lzm short
dflash: pass missing NVFP4 scales to attention operations (#28000) DFlash2 NVFP4 draft models produced almost no accepted speculative tokens because the Q, K, V, and output projection scales were not
hip : optimize Q2_0 dot-product path for gfx1201 (#26753) hip/gfx1201: optimize q2_0 vec_dot_q2_0_q8_1 with native amdgcn perm Broadened HIP's Q2_0 perm optimization Remove redundant HIP perm availabi
CUDA: use the fast mm_ids_helper path for any n_expert_used (#27978) The optimized path grouped warp lanes by token and required warp_size % n_expert_used == 0, with a single hardcoded exception paddi
llama: improve TENSOR_READ_LAZY handling (#27837) force lazy tensor on cpu if lazy is on llama: improve TENSOR_READ_LAZY handling Website: https://llama.app Attestations: https://github.com/ggml-org/l
vulkan: Change mul_mat_id to pad K rather than N (#27925) The N padding is needed for mul_mat, but not mul_mat_id. For mul_mat_id, we indirect the row index through a shared memory lookup table which
vulkan: combine duplicated fastdiv functions, rename the one optimizing small divs (#27526) vulkan: combine duplicated fastdiv functions, rename the one optimizing small divs remove one more fastdiv W
sycl: make --fit respect --fit-target better (#27629) improve the --fit algorithm to take into account the actual peak required VRAM for a given context size on a SYCL backend. This includes both prop
metal : assert shared memory padding (#27951) metal : assert shared memory padding cont : add ref Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/43867822 m
opencl: use a better matmul path on two Adreno GPU generations (#27640) opencl: default the Adreno xmem F16xF32 GEMM on for X2E kernel_mul_mm_f16_f32_l4_lm is the slowest matmul this backend has on Ad
model: add DSpark support for Nemotron3.5 (#27804) model: add DSpark support for Nemotron3.5 Update src/models/dflash.cpp Co-authored-by: Sigbjørn Skjæret sigbjorn.skjaeret@huggingface.co Co-authored-
tests : run test-save-load-state across all architectures (#27755) tests : run test-save-load-state across all architectures test-save-load-state previously only ran in ctest against a single download
sycl: bind the f16 KV cache in place for the oneDNN SDPA path (#27468) Measured at a live KV length of 34816 (32768 depth plus one 2048 ubatch), on Qwen3.8 27B Q4_K_S: per tensor 4 * 34816 * 256 * 2 B
sycl: use TILE for quantized KV decode on BMG (#26689) Route quantized KV decode to TILE on Xe2 (BMG) only, keep VEC on other archs until validated there. Website: https://llama.app Attestations: http
OpenVINO: Update OV to 2026.3.1, whisper.cpp support, Qwen3.5 on NPU, and new ops (#27843) OpenVINO Backend: Fuse IM2COL + MatMul convolution into OpenVINO convolution ci:ggml-ov: Skip recurrent state
Vulkan: add hoisting support for row IDs and expert count in shaders (#26686) vulkan: add hoisting support for row IDs and expert count in shaders use hoisted row ids in coopmat2 vulkan: address revie
ggml : fix conv_transpose_2d for multiple batches (#26132) ggml : fix conv_transpose_2d for multiple batches ggml_compute_forward_conv_transpose_2d_impl only computed the first batch (ne[3] of the des
vulkan: fix missing view-alias dependencies in ggml_vk_graph_optimize (#27812) vulkan: fix missing view-alias dependencies in ggml_vk_graph_optimize is_src_of doesn't treat two views of one tensor as
bench: add --tensor-read-lazy (#27881) bench: add --tensor-read-lazy rm the alias rename to LLAMA_LAZY_MODE_* Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestation
Hey guys, FastVideo team here. We saw how important speed and quality is for everyone. And we've been working to create our own step distill checkpoints and LORAs for MINIMAX h3. Here's is our v1 rele
hexagon: support for multi-NPU devices (IQ9, IQ10) and fully asynchronous backend (#26501) hexagon: use non-host bufs by default and make the backend fully async hex-hb: remove optional hostbuf suppor
models : support nanbeige4.2-3B (#27730) Co-authored-by: admin lizongqiang@kanzhun.com Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/43308396 macOS/iOS: m
llama : add --n-cpu-ffn option (#26622) common : dedupe --n-cpu-moe / --spec-draft-n-cpu-moe override loops common : add --n-cpu-ffn to CPU-offload dense FFN weights of first N layers common : general
args: add --video-* CLI arguments (#24318) args: add --video-* CLI arguments gen docs nits add mtmd_helper_init_opt Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attes
model : simplify MiniMax-01 graph (#27790) Co-authored-by: Stanisław Szymczyk sszymczy@gmail.com Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/43387702 ma
spec: Add benchmark-only synthetic speculative acceptance options (#27711) Add benchmark-only synthetic speculative acceptance to llama-server and llama-cli Address review comments Address review comm
llama: model_loader: add TENSOR_READ_LAZY (#27794) llama: model_loader: add TENSOR_GET_ROW_LAZY add --tensor-read-lazy rename to TENSOR_READ_LAZY gen docs address comments Website: https://llama.app A
Feature: Added LIGHTNING_INDEXER support for Deepseek V4 ops on Vulkan Backend (#27453) vulkan: add LIGHTNING_INDEXER op vulkan: updated lightning_indexer.comp and ggml-vulkan.cpp with 128-lane dot-pr
quantize: cap working memory size to avoid loading big tensors onto RAM (#27795) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/43468853 macOS/iOS: macOS A
opencl: add bin kernels kernel_gemm_moe_q4_0_q8_1_dp4a_bin, kernel_gemm_moe_mxfp4_q8_1_dp4a_bin (#27768) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/434
spec : add DFlash2 support (local convolution + candidate selector) (#27342) (#27816) spec : add DFlash2 support (local convolution + candidate selector) (#27342) support DFlash2 Add p_min in DFlash2
ci : bundle HIP runtime DLLs with Windows ROCm release (#26973) Copy amdhip64_7, amd_comgr and rocm_kpack next to the binaries so the correct HIP runtime loads over the driver's copy in System32. Fixe
model: add Qwen3.8-Flash-Next (qwen4exp) (#27742) gguf: add qwen4exp (Qwen3.8-Flash-Next) arch and converter Adds the GGUF-side plumbing for HF model_type qwen4_exp: MODEL_ARCH.QWEN4EXP plus tensors f
server: add ctx-per-slot (--kv-unified-per-slot) (#24124) Add ctx-per-slot argument for unifid KV cache Swap out ctx fractions for ctx pool slots Formatting cleanup Remove ctx-pool-slots, make ctx-per
hex-unary: fix RMS_NORM_MUL weight-offset bugs for grouped/broadcast norms (#27798) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/43526462 macOS/iOS: macO
ggml-hexagon: add HTP unary ops for ABS and LOG (#27786) Add HVX-accelerated implementations for GGML_OP_LOG and GGML_UNARY_OP_ABS on the HTP backend. Register HTP_OP_UNARY_ABS and HTP_OP_UNARY_LOG in
Got the new Qwen3.8-Flash-Next (176B total, 6B active MoE) running locally and figured I'd share for anyone else wondering if their setup can handle it. **My hardware:** 192GB DDR5 RAM, RTX 5070 Ti (1
arXiv:2608.25523v1 Announce Type: new Abstract: Prefix caching introduces a fundamental tradeoff in multi-agent large language model (LLM) serving: retaining a long system-prompt key-value (KV) cache
ggml-metal: add chunked SSD MMA for Mamba-2 prefill optimization (#26647) metal: WIP chunked SSD SSM_SCAN kernels for multi-token prefill metal: drop scalar SSD path; MMA + sequential tail drop WIP ss
cuda: unblock mmq for MoE on sm_60 (#26264) cuda: unblock mmq for MoE on sm_60 cuda: duplicate mmq-config-pascal for dp4a and older cuda: reduce occupancy on non-dp4a pascal for Q2_K, Q4_K, Q5_K, Q6_K
ci: Clean up UI builds from releases (#27706) ci : inline UI version resolution into ui-build.yml ci : build UI once and reuse the artifact in release jobs Server jobs now extract the ui-build artifac
vulkan: warptiles currently assume warp sizes <= 64, clamp to work around larger warps (#27726) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/43173045 mac
llama: add token ID tracking to KV cell (#27762) kv: track token id rm get_prev_tokens, move it to the main pr nits add get_prev_tokens Website: https://llama.app Attestations: https://github.com/ggml
grammar : parse - in char classes as literal hyphen (#27591) grammar : accept '-' escape in character classes gbnf_escape_char_class() escapes '-' as '-' but parse_char() rejected that escape, so gene
llama.cpp : bump version to 0.3.0 (#27696) llama.cpp : bump version to 0.3.0 ci : update release default desc scripts : add prompt for generating release summary Website: https://llama.app Attestation
metal : null-check buffer alloc to fix OOM crash (#25371) metal : null-check ggml_metal_buffer_init result to avoid OOM crash ggml_backend_metal_buffer_type_alloc_buffer used the result of ggml_metal_
ci : update OS used for ROCM to Ubuntu 24.04 (#27681) This matches what other build targets use and also what AMD advertises wheels as supporting. Website: https://llama.app Attestations: https://gith
kleidiai: Rework KleidiAI Build System/Integration (#26077) Rework KleidiAI Build System/Integration Signed-off-by: Jonathan Clohessy Jonathan.Clohessy@arm.com Add fp16 guard, and fix cmake caching is
arXiv:2503.18484v3 Announce Type: replace-cross Abstract: Evaluating the multilingual capabilities of Large Vision-Language Models (LVLMs) remains challenging because most benchmarks rely on non-paral
I made a Discord for people who are genuinely into AI and want a decent place to talk about it. It’s still new, but the idea is to build a large community without arbitrary bans, hidden moderation dec
Deepseek 4: -sm tensor (#26490) DSV4: sm tensor set coarser granularity for head splits fix dspark add model saving for dsv4 + allow dflash to return on specific device add comment about dsv4 seq_rm s
mamba2 : Flatten in/out projections to dispatch GEMM instead of GEMV (#27513) mamba2 : flatten mamba2 in/out projections to dispatch gemm instead of gemv mamba2 : remove redundant output reshape Websi
mtmd: video: fix moov atom at the end of file (#27596) mtmd: video: fix moov at the end of file Co-authored-by: rkfg rkfg@rkfg.me fix SIGPIPE windows: handle broken pipe case Co-authored-by: rkfg rkfg
ggml : shorten virtual device naming in CUDA and Metal (#27608) ggml : shorten virtual device naming in CUDA and Metal Assisted-by: llama.cpp:DeepSeek-V4-Flash-0731 ggml-metal : build device descripti
tests : disable DOTS3NOTE arch test for WebGPU (#27654) Co-authored-by: Stanisław Szymczyk sszymczy@gmail.com Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestation
metal: per-op source split + parallel compile (#26561) metal : per-op source split + parallel compile (#24021) preliminary extract common header op source split split metallib into 8 libs && load in p
metal : per-device tuned (Q, NE) for flash-attn vec (#26570) metal : per-device tuned (Q, NE) for flash-attn vec (#25750) rebase Q-generic FA vec body from 01dc936 (#23114) add 53 f16 (Q,NE) flash-att