Search: “apple”

jinja : add support for dict builtin (#29477) add support for dict builtin add tests Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50563209 macOS/iOS: mac

common : make string_split throw on invalid values (#29518) Signed-off-by: Adrien Gallouët angt@huggingface.co Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestatio

common : avoid side effects around params parsing (#29537) register --rpc unconditionally and call llama_supports_rpc() only from its handler print server 'initialization ...' log after args are parse

server : allow RANK pooling batch splitting for causal LLM rerankers (ie. Qwen3 and Qwen3-VL) (#28876) server : allow splitting RANK pooling for causal LLM rerankers Rerank models fall into two catego

Early this year when I was first looking at building up my inference capability you could get the 16GB Tesla P100s for between 60 and 80. Asked claude about it, told me absolutely not worth it. No ten

Good day data nerds! I am trying to get more into running models, learning about agentic workflows, and creating my own tools. But as you do (right?) I had to fine tune my current setup. With the way

Hi! I'm the author of MoEspresso, which is my way of putting my own ideas about inference engines to the test. A lot of the fun has been trying different design choices, measuring what happens, and fi

hexagon: find software divide calls using binary inspection tool (#29449) hex-scripts: fix table alignment hex-scripts: find sw div calls using binary inspection tool Website: https://llama.app Attest

ggml-cpu: tiled mul_mat for k-quants (#27851) Added tiled mul_mat. For each mul_mat_one_chunk, quants are unpacked into (max) 256x256 tiles of int8, one routine per quent. Then microkernel computes 16

Revert 'Change max context length for auto-fitting with unified KV (#28849)' (#29437) This reverts commit b04d4e5. Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attest

server : fix wake_fd warning on Windows (#29479) Signed-off-by: Adrien Gallouët angt@huggingface.co Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50455005

cuda: add F16 input to the FWHT (#29096) cuda: add F16 input to the FWHT The CUDA FWHT accepts F32 input only. This makes the source type a template parameter, so the kernel reads an F16 source direct

metal : fix graph capture and handle empty graphs (#29390) return early when the graph has no nodes drop the redundant reset of capture_compute: the decrement at the top of the function already transi

hexagon: use DMA for contiguous dim1 CONCAT (#29404) hexagon: use DMA for contiguous dim1 CONCAT Assisted-by: OpenCode hexagon: update CONCAT DMA for DMA64 Assisted-by: OpenCode Website: https://llama

CUDA: fuse RMS_NORM + SCALE into one kernel (#29393) #28068 builds the GDN q/k l2norm as ggml_scale(ggml_rms_norm(x, eps/n), 1/sqrt(n)). This adds 2 SCALE nodes per GDN layer, 96 extra kernel launches

musa: fix PH1 (MTT S5000) operator failures and build issues (#29193) musa: use 16-byte copies for MUSA like sm_70+ ggml_cuda_get_max_cpy_bytes() derives the copy width from CUDA_ARCH. mcc never defin

[SYCL] support sparse FA (#28796) fix conflict fix format issue rm unused code Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50188435 macOS/iOS: macOS App

rpc: include nb in the get_alloc_size cache key and floor the result at ggml_nbytes (#29283) rpc : include nb in the get_alloc_size cache key and floor the result at ggml_nbytes cont : remove redundan

HIP: bump HIP_VERSION requried for fp8 to avoid missing __hip_fp8_e4m3 support in 6.2 (#29231) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/50216799 macO

llama : add llama_prec_policy + model-driven W4A4 path (#24364) Rebase and update based on #26675 Signed-off-by: ynankani ynankani@nvidia.com CI failure fix(launh_bounds overload on HIP) and cleanup S

metal : split fa kernels into per-dtype libraries (#29329) metal : split fa kernels into per-dtype libraries Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp cont : minor fix comment Website: ht

metal: FWHT kernels for block widths above 512 (#29095) metal: FWHT kernels for block widths above 512 The Metal FWHT covers widths 64 to 512, one row per simdgroup with N/32 values per lane. Wider bl

common : extract shared unicode path/string helpers (#29415) Signed-off-by: Adrien Gallouët angt@huggingface.co Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestati

Fixing the vulkan build issue of legacy GLSLC version that has no cooperativeMatrix API support (#29373) (#29409) vulkan : fix build issue of legacy glslc version by adding GGML_VULKAN_COOPMAT_GLSLC_S

opencl: add bin kernel kernel_gemm_noshuffle_q5_k_f32_32b_trans_ila_a8_bin, kernel_gemm_noshuffle_q5_k_q8_1_dp4a_ila_a8_bin (#29401) opencl: add A8 Q5_K non-MoE non dp4a + dp4a binary kernel opencl: f

mtmd: fix mel preprocessor in LFM2 audio (#29403) which resulted in different greedy transcripts for 4.5% of English and 6.5% of Japanese test utterances. In Japanese, some differences changed entire

common,rpc : simplify fs_create_directory_with_parents() (#29432) The original function was broken on Windows for some unicode paths Paths without a trailing separator now create the last directory to

For those who have V100 cards, I wanted to point you to 1Cat-vLLM, a vLLM fork that enables optimized serving for these cards. Showing stats for Qwen3.6-35b comparing a Strix Halo with a hughly optimi

FYI the engine's default kernel rules were measured on smaller chips (16/20-core M5s and a 32-core M4 Max), so a 40-core M5 Max runs guesses. Splash's repo includes a developer tool, “make tune-kernel

test-save-load-state : print a per-model results table in --models mode (#29316) test-save-load-state : print a per-model results table in --models mode in --models mode the output was very heavy: eve

server,common : fix the GCC 12 stringop-overread false positive (again) (#29325) Signed-off-by: Adrien Gallouët angt@huggingface.co Website: https://llama.app Attestations: https://github.com/ggml-org

model : add Ling 3.0 VL support (#29151) model : fold Ling 3.0 VL into the BailingMoeV3 architecture Assisted-by: Scout model : keep shared NORM rope list intact when gating bailingmoe3 on mrope secti

cuda : add conv3d with implicit GEMM (#29137) cuda : add conv3d with implicit GEMM cuda : refine conv3d implicit GEMM and handle empty kernels Website: https://llama.app Attestations: https://github.c

vulkan: tune KHR cooperative matrix support for Adreno GPUs (#29328) Enable coopmat support for Vulkan backend Fixed the mul_mat_s Removed the debug statement Website: https://llama.app Attestations:

vulkan: handle misalignment in conv_2d and conv_3d (#29365) vulkan: handle misalignment in conv_2d and conv_3d fix test-backend-ops print Website: https://llama.app Attestations: https://github.com/gg

vulkan: int8 coopmat1 matmul implementation for AMD RDNA3 and RDNA4 (#27952) vulkan: add int8 coopmat quantized matmul shader apply scales inline use scalar sums probe and directly access coopmat valu

llama: add llama_batch_ext (#24669) (wip) add llama_batch_ext wip updated design updated impl change signature unused var demo common_prompt_batch_decode fix pos tmp disable test-batch-alloc fix compa

hexagon: dynamic quantizer improvements (#29395) hexagon: fix accuracy issue in Q8_0 N=1 MUL_MAT hex-quant: fix register spills hex-mm: use dma for all dyn.quant paths Co-authored-by: Aparna M P aparm

sync : ggml (#29396) ggml : bump version to 0.25.2 (ggml/1642) ggml : fix ubsan error in ggml_graph_nbytes (ggml/1644) ggml : bump version to 0.25.3 (ggml/1645) sync : ggml Website: https://llama.app

metal : optimize sparse FA + clean-up (#29377) metal : cache sparse FA indices in shared memory Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp metal : simplify shared memory size calculation A

Fentanyl Grade Compute: Why a GB300 Rack Will Out-Price Blow by Weight in 2030 A GB300 NVL72 weighs roughly 1,580 kg fully populated and recent purchase orders put it at 5M per rack. That is 3,165/kg.

I'm about to buy a Mac Studio mainly for running LLMs locally and I'm stuck between two configs: M5 Ultra (30/64) with 96GB: 1.2 TB/s bandwidth, roughly 1.7x faster generation and much faster prefill

Hey everyone, Jovan from UkisAI here! Today, we are introducing Swift, a family of efficient reasoning LLMs based on Qwen, trained by penalizing tokens related to pathological overthinking patterns an

HIP : optimize IQ2/IQ3 (__vsub4 __vcmpne4) using SWAR (#27962) HIP : use bit manipulation for __vcmpne4 HIP : use bit manipulation for __vsub4 Website: https://llama.app Attestations: https://github.c

hex-dma: introduce direct-mapped DMA cache that is better suited for HVX FA mask handling (#29282) Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/49404271

vulkan: hide internal symbols to prevent duplicate-dlopen state destruction (#29139) Since #28732 our internal symbols are exported. A duplicate copy dlopened and dlclosed by ggml_backend_load_all() t