b10909

metal : single-source fusion table + fusion debug rework (#28164) metal : rework fusion patterns into a single table All fusable op patterns for the Metal backend are now declared once in a fusion table (ggml-metal-fuse.cpp) and consumed by both the graph optimizer (ggml_metal_fuse_max, packing) and the op encoders (ggml_metal_fuse_next, compute). The two phases share the same pattern table plus ggml_can_fuse_subgraph_ext for the structural checks, and differ only in the mode used for the pattern check (STRUCTURAL at optimize time, since tensors are not allocated yet, and FULL at compute time, including Metal buffer placement). This also protects the snake activation (MUL + SIN + SQR + MUL + ADD) from being reordered during graph optimization, which was previously unprotected. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 metal : fix absolute output indices in fusion patterns ggml_can_fuse_subgraph_ext expects the outputs array to contain absolute graph node indices (it indexes cgraph->nodes[outputs[i]]), but the fusion table query was passing a relative index (n_ops - 1). As a result the last node of every pattern was not recognized as an output and was subjected to the elidable use-count check, which failed for essentially all fusions. This silently disabled the norm/MUL fusion and caused a ~5% token-generation regression. Pass the absolute graph index of the last node instead. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 metal : fuse gated_delta_net with cache cpy Add GGML_METAL_FUSE_GDN_CACHE to the fusion table: when the gated_delta_net kernel is followed by a cpy that scatters its recurrent state snapshots into the KV cache, the kernel writes the snapshots straight into the cache buffer and the trailing cpy is elided. The gdn output has other consumers (the attn scores view), so unlike the elision-chain patterns this is not a simple chain: a 'raw' flag on the fusion pattern skips the generic chain/shape and ggml_can_fuse_subgraph_ext checks, making the pattern-specific check callback the sole validator. Packing (ggml_metal_fuse_max) now matches on the same view-transparent node sequence that the compute phase uses, so the gdn + cache cpy group is packed along with any intermediate views and stays adjacent through the reorder. The fused cpy is a view consumer of the gdn (it writes the cache directly), so its mem-range is skipped in the encoder; the skip is restricted to CPY nodes consuming the previous fused node through a view so other fusions are unaffected. Add test_gated_delta_net_cache_fusion and register 5 cases. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 metal : drop is_view_consumer mem-range skip The is_view_consumer skip was carried over from the upstream gated_delta_net cache-fusion draft, but it is not needed: keeping the elided cpy's mem-range in the concurrency tracker only ever adds a (conservative) memory barrier at the fusion point. It can never remove a barrier, so it cannot introduce a race. The worst case is one spurious barrier per gdn+cache-cpy fusion, which is within run-to-run noise on Qwen3.5-0.8B Q8_0. Dropping the check keeps the mem-range loop uniform for all fused groups. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 metal : rename gated_delta_net fused state output args Rename the fused cache-write kernel argument to match the rest of the kargs: state_out_stride -> nb_out (and widen it to uint64_t), and the local buffer id bid_state_out -> bid_out. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 metal : rename raw fusion flag to unsafe raw did not convey that the flag opts a fusion pattern out of the generic elision-chain safety net (ggml_can_fuse_subgraph_ext + chain/shape checks). rename it to 'unsafe' to make explicit that the pattern's check callback is the sole validator and must re-establish the safety guarantees itself. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 metal : tidy fusion pattern checks and table const-correct ggml_metal_fuse_outputs buffer annotate unused check-callback parameters drop a redundant size_t cast align the ops/table initializers and add blank-line separation Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 metal : add generic fusion stats via ad-hoc proc-address API Add a device-owned fusion context that lets a test tool count how many times each fusion pattern fires and toggle fusion. It is exposed through the ad-hoc ggml_backend_reg_get_proc_address mechanism with generic names so the testing tool is backend-agnostic: ggml_backend_fusion_stats_init: start collecting fusion stats; when a context is created afterwards it registers the labels/counters and encodes single-threaded (n_cb == 0) so the counters are race-free ggml_backend_fusion_stats_reset / _get_stats / set_enabled The context lives on the metal device (not on the last backend context), so counters accumulate across contexts and reads are always consistent. The enable/disable toggle is initialized from GGML_METAL_FUSION_DISABLE and can be overridden by the test through set_enabled. Labels are synthesized from the fuse table via ggml_metal_fuse_label (e.g. "GATED_DELTA_NET+CPY"). Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 tests : add fusion count regression test with per-backend baseline test-fusion runs every dummy model generated by test-llama-archs on a single backend (single-threaded encoding, n_cb == 0) with fusion enabled and disabled, and for each mode (prefill / decode) reports the per-fusion counters and the NMSE between the fused and unfused logits, plus the NMSE against a CPU reference. A fusion pattern that silently stops matching (or fires when it should not) is caught as a regression by comparing the counters against a committed per-backend TSV baseline: --record writes the golden baseline, --check (default) validates it the unfused run doubles as a control: its counters must be all-zero NMSE is skipped when it is NaN or the arch is already broken on the device (e.g. plamo2 on Metal), so the count check is the hard gate baseline counts depend only on graph structure, not weights (verified stable across weight seeds) the fusion stats API is resolved through the ad-hoc get_proc_address mechanism with generic names; a backend that does not export it makes the test fail with an error The committed MTL0.tsv baseline covers 110 dummy archs (298 rows). Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 tests : rename fusion api helpers to match stats_init signature Align the test with the ad-hoc fusion stats API: fusion_stats_init no longer takes an enable bool (stats are turned on by calling it), so the proc-address wrappers and typedefs are renamed to the api* convention. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 tests : rename backend to device in fusion test CLI The fusion test operates on a compute device (e.g. MTL0), not a backend, so rename the --backend argument to --device and the backend_name variable to device_name. Keep "backend" where it refers to the ggml backend interface (the ad-hoc proc-address mechanism). Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 tests : add --model and --help to fusion test --model FILE runs the fusion regression test over a single model file instead of enumerating a --models DIR. --models and --model are mutually exclusive. Also add a --help/-h option that prints the usage. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 tests : use backend base name for fusion baseline output The fusion test is invoked with a specific device name (e.g. MTL0), but its output - the recorded baseline and the header it writes - should be named after the backend base name (e.g. MTL, via ggml_backend_reg_name), since the counters depend on the backend, not on the specific device index. Rename the committed baseline to MTL.tsv. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 tests : run fusion test from ci instead of ctest The fusion test needs Metal and generates a lot of dummy models, so it does not belong in the generic ctest suite. Move it to ci/run.sh as gg_run_test_fusion, gated on GG_BUILD_METAL like gg_run_test_llama_archs_tensor_split: it generates the dummy models with test-llama-archs -o and then validates the fusion counts against the committed baseline. test-fusion.cpp is still built (llama_build) but no longer registered as a ctest. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 tests : align fusion baseline TSV columns Pad the TSV fields to fixed widths so the columns line up regardless of the variable arch and fusion-label lengths, and trim each field on parse so the padded file is still accepted. Regenerate the committed MTL.tsv baseline in the padded format (data unchanged, verified identical modulo padding). Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 tests : widen label column and align fusion TSV header Give the label column more room (28 chars) and fix the column header widths so they match the data rows (moe/mode/label), keeping the header aligned with the values. Regenerate the MTL.tsv baseline in the new format (data unchanged). Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 tests : switch fusion baseline from TSV to CSV Use comma-separated values like the rest of the project, keeping the padded, aligned columns. Split on ',' and trim on parse. Rename the committed baseline to MTL.csv (data unchanged, verified identical modulo padding/separator). Update the ci/run.sh check path accordingly. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 cont : rebase + update MTL stats tests : avoid graph reallocations for some archs metal : tidy fusion debugging context and op init simplify the shared fusion debugging context comments shorten the ggml_metal_fusion struct comment align the ggml_metal_fuse struct fields and comments move the fusion parameter of ggml_metal_op_init right after dev Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 tests : dedup fusion baseline into any mode prefill and decode always produce the same per-graph fusion count, so store a single row per label with mode = "any" and the per-graph count instead of two rows. this halves the baseline size and keeps the check stable. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 ci : move fusion model generation to a separate step the dummy models generated by test-llama-archs are reused by other tests, so generate them once in their own step instead of inside test_fusion. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 tests : bump nmse thold models : fix plamo2 graph tests : remove "skip" logic from test-fusion tests : set qwen3tts dummy vocab to codec head size the dummy qwen3tts model used a vocab of 4096 while the codec head is 3072, so the graph padded the output with -inf which made the NMSE in test-fusion produce NaN. use the exact codec head size instead so the padding is not generated at all. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 tests : regen fusion baseline reflect the plamo2 graph fix, which changed its fusion pattern split (RMS_NORM+MUL 11->10, RMS_NORM+MUL+ADD 3->4; same total). Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 ci : skip dummy model generation on OpenVINO test-llama-archs does not build on the OpenVINO platform, so do not try to generate the dummy models there. Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 cont : minor tests : enable test-llama-archs on windows cont : disable on windows + workaround metal : naming nits test-fusion : add instructions to update baseline context : fix Kimi-K3 graph reserve fusion : update MTL cont : fix naming metal : rework fusion info storage Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp metal : align fusion info API Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp metal : use opaque fusion handle in ad-hoc API Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp ci : move fusion test to dedicated workflow Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-Vision-Exp cont : run only on ggml changes cont : simplify fusion : remove multi-output stuff for now ci : fix typo Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/46845298 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan) Ubuntu arm64 (Vulkan) Ubuntu x64 (ROCm 10.0) Ubuntu x64 (OpenVINO) Ubuntu x64 (SYCL FP32) Ubuntu x64 (SYCL FP16) Android: Android arm64 (CPU) Windows: Windows x64 (CPU) Windows arm64 (CPU) Windows arm64 (OpenCL Adreno) Windows x64 (CUDA 12) - CUDA 12.4 DLLs Windows x64 (CUDA 13) - CUDA 13.3 DLLs Windows arm64 (CUDA 13) - CUDA 13.4 DLLs Windows x64 (Vulkan) Windows x64 (OpenVINO) Windows x64 (SYCL) Windows x64 (ROCm 10.0) openEuler: DISABLED openEuler x86 (310p) openEuler x86 (910b, ACL Graph) openEuler aarch64 (310p) openEuler aarch64 (910b, ACL Graph) UI: UI

Source: llama.cpp Releases | 2026-09-11

Related