b10776

model: add NVIDIA Nemotron-3-Puzzle-75B-A9B (NemotronHPuzzle) support (#25444) hparams: add per-layer n_ff_exp/n_expert_used arrays with scalar-or-array loading G1/G2 infrastructure for variable-per-layer expert FFN size and top-k routing (required for Puzzle-75B which has 5 distinct n_ff_exp values and 7 top-k values across its 40 MoE layers). Design: rename scalar members to impl suffix (following existing convention), add LLAMA_MAX_LAYERS arrays, add n_ff_exp(il)/n_expert_used(il) accessors with scalar fallback. No new GGUF keys: reuses existing expert_feed_forward_length and expert_used_count keys via get_key_or_arr (scalar -> broadcast, array -> per-layer). llama-hparams.h: n_ff_exp -> n_ff_exp_impl, n_expert_used -> n_expert_used_impl; add n_ff_exp_arr / n_expert_used_arr arrays; add per-layer accessor declarations. llama-hparams.cpp: implement n_ff_exp(il) and n_expert_used(il); out-of-range il returns impl safely (shared code, no abort). llama-model.cpp: central n_expert_used load changed to get_key_or_arr; derive impl as max-of-array for validations and backward compat; zero both new arrays; HunyuanVL override also zeroes n_expert_used_arr. llama-graph.cpp: aggregation loop in build_moe_ffn uses hparams.n_expert_used(il) so per-layer top-k bounds the ggml_view loop correctly. All other files: mechanical rename hparams.n{ff_exp,expert_used} -> _impl. Scalar arches are unaffected (broadcast fills all array slots with the single value). (cherry picked from commit 269a81e) nemotron-h: use per-layer n_ff_exp(il) and n_expert_used(il) at MoE call-sites Load n_ff_exp via get_key_or_arr into hparams.n_ff_exp_arr in load_arch_hparams; derive impl as max for existing uniform GGUFs. In load_arch_tensors, compute n_ff_exp_i = hparams.n_ff_exp(i) with fallback to n_ff(i)/n_expert_used(i) for GGUFs that omit expert_feed_forward_length. In build_ffn_layer, pass hparams.n_expert_used(il) to build_moe_ffn so per-layer top-k is used for expert routing selection. All other nemotron-h behaviour (mamba2, attention, shared-exp, latent projection, routed_scaling_factor, expert_weights_norm, sigmoid gating) is unchanged. (cherry picked from commit b1878a1) arch/.cpp + gguf-py: mechanical rename n_ff_exp->n_ff_exp_impl, n_expert_used->n_expert_used_impl All non-nemotron arch files continue using the scalar impl member directly. Behaviour is identical: the impl value is the broadcast value from the GGUF scalar. gguf_writer: add_expert_feed_forward_length and add_expert_used_count now accept int | Sequence[int], mirroring add_feed_forward_length, so converters can write per-layer arrays with the same existing GGUF keys. (cherry picked from commit 8f009f5) convert: support NemotronHPuzzleForCausalLM (per-block MoE config) Parse block_configs/mtp_block_configs into per-layer arrays (scalar-or-array keys), append the MTP [attention, moe] sub-blocks as blk.88/blk.89 with nextn tensors, accept the backbone.* prefix, and register the arch. Also fix a pre-existing undeclared _experts attribute on NemotronHModel. (cherry picked from commit d1a592f) nemotron-h: distinguish Nemotron 3 Puzzle (75B.A9B) from Super (120B.A12B) Both have 88 layers; the per-layer expert_used_count array (heterogeneous for Puzzle, broadcast-uniform for Super) is the discriminator. (cherry picked from commit f824e09) convert: accept the official Puzzle BF16 checkpoint's tensor naming The officially distributed BF16 checkpoint (NVIDIA-Nemotron-Labs-3-Puzzle- 75B-A9B-BF16) names the trunk model.* (model.layers., model.embeddings, model.norm_f) where the original release used the NemotronH-style backbone., and spells the router bias e_score_correction_bias instead of e_score_correction.bias. Normalize both at the top of NemotronHPuzzleModel.modify_tensors so either checkpoint converts; every tensor name in the official index (42683 keys, MTP head included) resolves through the tensor map after normalization. (cherry picked from commit 189b67f) laguna: use n_ff_exp_impl for the uniform-MoE FFN size Laguna landed after this branch was cut and reads hparams.n_ff_exp as a scalar. This series turns it into a per-layer array with an n_ff_exp(il) accessor, so the three scalar reads no longer compile. Laguna is a uniform MoE, so point them at the scalar fallback n_ff_exp_impl, same as deepseek2/qwen3moe/gemma4 in this series. No behaviour change. (cherry picked from commit dbedc9e) arch: extend the n_ff_exp/n_expert_used rename to archs added upstream kimi-k3, dflash, bailingmoe3, deepseek4, granite-swa and the nemotron-h MTP block still referenced the scalar fields by their old names. n_ff_exp and n_expert_used are accessors now, so those reads no longer compile; point the non-per-layer archs at the _impl scalars and use the indexed form where the call site is per-layer. convert: keep Puzzle opted out of the NemotronH MTP export path #26725 added MTP export to NemotronHModel, keyed on num_nextn_predict_layers. Puzzle's config carries that key, but NemotronHPuzzleModel bypasses NemotronHModel.init (its per-block config needs a different setup), so _mtp_bid was never assigned and modify_tensors raised AttributeError on any mtp.* tensor. Puzzle's head is also laid out by mtp_block_configs, not the mtp.layers.* form the base maps. Set _mtp_bid to None, drop mtp.* in filter_tensors, and declare supports_mtp_export = False so --mtp / --no-mtp fail at the CLI. llama: replace n_ff_exp/n_expert_used scalars with per-layer accessors Follow-up to review feedback: the previous revision kept the scalar hparams fields alongside the new per-layer arrays, which duplicated state that get_key_or_arr already handles by broadcasting a scalar value over every layer. Drop both scalars and expose n_ff_exp(il) / n_expert_used(il) built exactly like the existing n_head_kv(il) and n_ff(il) accessors: they index the array and GGML_ABORT out of range, with il defaulting to 0 so genuinely uniform call sites stay a plain n_ff_exp(). Arch loaders now read both keys through get_key_or_arr over n_layer_all, and the n_expert_used validation checks the maximum across layers instead of a single field. llama: restore per-key required flags on the expert hparam reads The scalar-to-array conversion passed required=false at every call site, which silently made mandatory keys optional. Each read now carries the same required flag it had before the conversion. Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/44926452 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan) Ubuntu arm64 (Vulkan) Ubuntu x64 (ROCm 10.0) Ubuntu x64 (OpenVINO) Ubuntu x64 (SYCL FP32) Ubuntu x64 (SYCL FP16) Android: Android arm64 (CPU) Windows: Windows x64 (CPU) Windows arm64 (CPU) Windows arm64 (OpenCL Adreno) Windows x64 (CUDA 12) - CUDA 12.4 DLLs Windows x64 (CUDA 13) - CUDA 13.3 DLLs Windows arm64 (CUDA 13) (preview) - CUDA 13.4 DLLs Windows x64 (Vulkan) Windows x64 (OpenVINO) Windows x64 (SYCL) Windows x64 (ROCm 10.0) openEuler: DISABLED openEuler x86 (310p) openEuler x86 (910b, ACL Graph) openEuler aarch64 (310p) openEuler aarch64 (910b, ACL Graph) UI: UI

Source: llama.cpp Releases | 2026-09-03