Model Releases
b11195
ggml-cpu: tiled mul_mat for k-quants (#27851) Added tiled mul_mat. For each mul_mat_one_chunk, quants are unpacked into (max) 256x256 tiles of int8, one routine per quent. Then microkernel computes 16
ggml-cpu: tiled mul_mat for k-quants (#27851) Added tiled mul_mat. For each mul_mat_one_chunk, quants are unpacked into (max) 256x256 tiles of int8, one routine per quent. Then microkernel computes 16x16 tiles before writing out 256x256 float reults to main memory. Tests/benches in tests/test-tiled-mulmat.cpp. 3-6x speed improvement for large matmul, break even at 4096x64 * 64x4096, 80% performance (net loss) for GEMV. Error rates trivial (order of 1-e04 max, 1-e05 rmse). Fixes for ARM/windows builds more windows fixes, ggml-cpu.h isn't visible in MSVC for some reason unified iqp + tiled on the Q5_K, IQ4_XS set for benchmarking, updated benchmark Fixed accidental removal of llama_build_and_test(test-backend-ops.cpp) First integration of iqp code Co-authored-by Bartowski 3266127+bartowski1182@users.noreply.github.com Cleaning up declaration of iq unpacking helpers to align with the bit unpackers Removed iqp path Fix cross-platform warnings Disabling benchmarks unless explicitly enabled Fix backend_init for DLL-based builds, add self and bartowski to CODEOWNERS for tiled Put benchmarks behind a flag kernel fix for AVX2, iq quants Fix for asan, leaking memory in test-tiled-mulmat and avoid stack use after return guarding env flags with std::call_once Simplified repacking for VNNI to a single call per macrotile No threadlocals anymore, aligned wdata access Doing aligned reads since we ensure alignment with padding in wdata Eliminated per-thread gather of Q8_K rows in mul_mat_id, we now gather/repack in a single pass. Repack method now takes pointer array to support both dense/normal and mmid paths. Interface with ggml-cpu.c simplified as a result Unified/simplified dispatch and support checks. Put details on wdata needed inside the kernel.h body, simplified interactions with ggml-cpu.c. Cleanup includes and whitespace, update src1_repack to return false if we don't need a special repack, so the common case is handled by driver Better detection of win32 and additional whitespace fixes Gating fuzz tests behind a parameter and some extra prints to try and fix slow CI hosts Optimized AVX2 kernel Changed interleave format and added ability to interleave in-place after dequant Repacks now happen in-place, 16x64 microtiles are independent of each other Only repack rows in groups of 16 as they're needed. Save work in low n_rows cases and optimize L1 usage in other cases Use long panels for memory-bound regime (M <= 16), reintroduce IQP path for benchmarks Fix unused warnings and cleanup. Improved IQ dequantization speed. Removed separate process benchmarks Revert "Removed separate process benchmarks" This reverts commit 0688cf4. AVX2 optimizations and guards for tests on windows Removed temp perf harness Remove perf-mulmat from build Removed IQP path, simplified tests to not use sub processes Cleaning up alignment of wdata Whitespace fixes and aligning L2 workspace to clean 512kb boundaries Update ggml/src/ggml-cpu/tiled/tiled-kernel.cpp Co-authored-by: Georgi Gerganov ggerganov@gmail.com Cleanup merge-duplicated declaration of test-backend-ops target Undo accidental line deletion in ggml.c Co-authored-by: Georgi Gerganov ggerganov@gmail.com Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50387125 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan) Ubuntu arm64 (Vulkan) Ubuntu x64 (CUDA 12) - CUDA 12.8 libraries Ubuntu x64 (CUDA 13) - CUDA 13.4 libraries Ubuntu arm64 (CUDA 13) - CUDA 13.4 libraries Ubuntu x64 (ROCm 10.0) Ubuntu x64 (OpenVINO) Ubuntu x64 (SYCL FP32) Ubuntu x64 (SYCL FP16) Linux arm64 (Snapdragon: CPU, Adreno GPU, Hexagon NPU) - setup guide Android: Android arm64 (CPU) Android arm64 (Snapdragon: CPU, Adreno GPU, Hexagon NPU) - setup guide Windows: Windows x64 (CPU) Windows arm64 (CPU) Windows arm64 (OpenCL Adreno) Windows x64 (CUDA 12) - CUDA 12.4 DLLs Windows x64 (CUDA 13) - CUDA 13.4 DLLs Windows arm64 (CUDA 13) - CUDA 13.4 DLLs Windows x64 (Vulkan) Windows x64 (OpenVINO) Windows x64 (SYCL) Windows x64 (ROCm 10.0) openEuler: DISABLED openEuler x86 (310p) openEuler x86 (910b, ACL Graph) openEuler aarch64 (310p) openEuler aarch64 (910b, ACL Graph) UI: UI
Related
Source: llama.cpp Releases | 2026-09-26