Model Releases
b11599: llama : do not map foreign ranges into a backend buffer (#30405)
llama : do not map foreign ranges into a backend buffer A buffer created from a host pointer pins everything that it wraps. When several contexts share the same file (lazy tensors, MoE tensors kept on
llama : do not map foreign ranges into a backend buffer A buffer created from a host pointer pins everything that it wraps. When several contexts share the same file (lazy tensors, MoE tensors kept on the CPU with -cmoe), the single buffer that spans the first and the last tensor of a context also wraps the ranges owned by the other contexts, pinning data that the backend never reads. For models with per-layer token embeddings this pins the whole lazy tensor into the GPU buffer (27 GiB for Qwen3.8-Flash-Next IQ4_NL), defeating the point of reading it lazily. Instead, compute the file ranges of each context and subtract the ranges owned by the other contexts, then create one buffer per range. Weights that are mapped in more than one context (e.g. token_embd.weight) keep being part of the range of every context that uses them. Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next llama : enable prefetch on macOS The prefetch hint of the tensors marked as prefetchable is a no-op on macOS, so their pages are faulted in one by one while the graph runs. madvise (MADV_WILLNEED) behaves as expected here, so add the platform to the guard. With the lazy per-layer token embeddings of Qwen3.8-Flash-Next IQ4_NL read from a cold file cache, prompt processing goes from 356 / 375 / 475 t/s to 735 / 749 / 788 t/s for pp512 / pp2048 / pp4096. Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next llama : use a plain buffer list per context Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
Related
Source: llama.cpp Releases | 2026-10-13