Ling-3.0-tiny is a very interesting model. Run on NVIDIA Orin Nano Super 8GB at 128K context with IQ4_NL quant.

I have been searching for suitable model to run on my 8GB RAM toy, NVIDIA Orin Nano Super 8GB. This little toy was priced at 249 earlier this year (not any more), and pulls very little power when idle. It was an interesting device that suitable for an agent to host on. It is likely won't do any coding job in meaningful way, but likely can handle simple things just fine. When the llama.cpp support showed up on github, I asked my Hermes to setup it up and run some basic test for me. The result is quite encouraging. Below is the summery from my Hermes run. ===============AI content below=================== # Ling-3.0-tiny at full 128K context on a 249 8GB Orin Nano Super — IQ4_NL, 33 tok/s, on llama.cpp master Tl;dr: I got inclusionAI's Ling-3.0-tiny (7.9B total, ~1.4B active MoE) running on an NVIDIA Jetson Orin Nano Super 8GB with the IQ4_NL quant at its full native 131,072-token context . Decode runs ~33 tok/s short-context (faster than you can read), prompt eval 220-760 tok/s, and the whole thing fits in 7.4 GiB unified RAM with only ~200 MB of swap. Model + KV + CUDA offload, all on one 249 board. ## The hardware - Jetson Orin Nano Super Developer Kit — official NVIDIA product page — was 249 at launch (price has risen since; SparkFun lists it at 399 now). 8 GB unified LPDDR5, 67 TOPS, JetPack R39.2 / CUDA 13.2. ## The model + quant - Original: inclusionAI/Ling-3.0-tiny — hybrid KDA+MLA reasoning MoE, 128 experts, 8 routed + 1 shared active per token - GGUF repo: bartowski/Ling-3.0-tiny-GGUF - File used: Ling-3.0-tiny-IQ4_NL.gguf — 4.40 GB, IQ4_NL (4.5 bpw), imatrix-calibrated - Why IQ4_NL: at 4.30 GiB it's the largest ≥4-bit quant that still leaves room for the full 131K KV window on 8 GB. Q5/Q6 fit only to ~64K; IQ4_NL goes the distance. ## The engine — this matters The stock quantizer release will not load this model . Ling-3.0-tiny is a BailingMoE V3 architecture, which only landed in llama.cpp via: - PR #26608 — BailingMoE3 Support (merged into master Aug 17, 2026, adds arch + speculative MTP support) - Branch: ggml-org/llama.cpp master , anything after that merge — clone fresh, don't use a release tarball - Built on-device with CUDA 13.2 for sm_87 (-DGGML_CUDA=on -DCMAKE_CUDA_ARCHITECTURES=87), all 24 layers offloaded Working command: llama-server -m Ling-3.0-tiny-IQ4_NL.gguf -c 131072 --cache-type-k q8_0 --cache-type-v q8_0 -ngl 99 -fa on -b 512 -ub 256 --jinja -t 6 --temp 1.0 --top-p 0.95 --top-k 20 ## Speed (measured on-device) | Test | Result | |---|---| | Prompt eval (2-8K) | 450-760 t/s | | Prompt eval (full 125K prefill) | 220-264 t/s | | Decode (short ctx) | 33 t/s | | Decode @ 96-128K depth | 15-17 t/s | | TTFT (512 tok) | ~1 s | Memory at 128K: 7.2/7.4 GiB used , ~200 MB swap. The hybrid design keeps the KV footprint small — only the 6 MLA layers carry context-proportional cache while the 18 KDA layers keep a fixed-size recurrent state — which is what makes the full window fit. ## Is it coherent at 128K? Mostly. I ran a proper needle-in-haystack: 128 unique passkeys embedded across the context, each queried individually at temp 0: - 96K context: 92% retrieval (118/128) — 100% in the first 32K - 128K context: 88% retrieval (113/128) — with a visible cliff in the final 32K (67%) - Strict full-string match is lower (~30-40%) because the model often emits just the tail chunk of the passkey — a generation artifact, not a retrieval miss - Failure modes in the deep zone: digit-mangling (needle 105 → answers 015's key) and a couple of confabulations So: majority passed, not a flawless 128K — but at 96K the cliff disappears entirely, and for a 249 board that's genuinely impressive capability-per-dollar. ## Verdict Worth spending more time on. The hybrid KDA/MLA architecture + IQ4_NL + llama.cpp master is a compelling recipe for edge deployment: agentic reasoning MoE, native 128K window, stable 15-33 tok/s, all in 8 GB unified memory. I'd like to see more folks poke at the 96-128K attention degradation — and whether the MTP spec-decode support from #26608 helps on-device. Anyone else running Ling-3.0-tiny on constrained hardware? What quant/config are you using? # Ling-3.0-tiny IQ4_NL — Orin Nano Super 8G vs RTX PRO 6000 Blackwell Same quant (IQ4_NL 4.5 bpw, 4.29 GiB), same llama.cpp master code (post-PR #26608 BailingMoE3), same flags (-ngl 99, q8_0 KV, flash-attn, -c 131072). ## Hardware | | Orin Nano Super 8G | RTX PRO 6000 Blackwell | |---|---|---| | GPU arch | sm_87 (Orin nvgpu) | sm_120 (Blackwell Max-Q) | | Memory | 7.4 GiB unified + 32G swap | 97.9 GiB VRAM | | Memory BW | ~102 GB/s | ~1.8 TB/s | | Launch price | 249 | ~7,000+ | ## Results | Test | Orin 8G (sm_87) | PRO 6000 (sm_120) | Speedup | |---|---|---|---| | pp512 | 451 t/s | 12,296 t/s | 27x | | pp2048 | 758 t/s | 17,435 t/s | 23x | | pp8192 | 722 t/s | 16,945 t/s | 23x | | pp96000 | 264 t/s | 8,093 t/s | 31x | | pp128000 | 220 t/s | 6,498 t/s | 30x | | tg32 | ~33 t/s | 286 t/s | ~9x | | tg128 (short ctx) | 33.2 t/s | 315 t/s | 9.5x | | tg512 | 33.4 t/s | 322 t/s | 9.6x | | tg128 @ 96K depth | 15-17 t/s | 318 t/s | ~19x | | tg128 @ 128K depth | 15-17 t/s | 314 t/s | ~19x | Sanity check: 17*23 = 391 correct on both. ## Key observations - Prefill scales ~30x (bandwidth-bound): Orin 102 GB/s vs Blackwell ~1.8 TB/s. A 125K-token prefill takes 9.7 min on the Orin vs ~19s on Blackwell. - Decode scales ~10x at short context but ~19x at full depth: the Orin's decode collapses from 33 to 15 t/s as KV depth grows (attention + memory pressure), while Blackwell stays flat at ~315 t/s regardless of depth (97 GB VRAM makes KV trivial). - MoE expert sparsity keeps decode fast on both: 7.9B total params with only ~1.4B active per token. - On Blackwell this model is effectively a background process: 322 t/s decode = 100-token response in 0.3s; 128K context costs it nothing. submitted by /u/Puzzleheaded_Base302 [link] [comments]

Source: r/LocalLLaMA | 2026-08-19