nvidias nemotron omni only loads its text half on a mac, so i wrote the vision and audio towers in mlx

nvidias nemotron omni is open weights and it sees, hears and reasons. theres already a 4bit mlx quant on hugging face but only the text backbone loads with standard mlx tooling. the model card says it plainly, the vision and audio towers need a runtime that implements the c-radio vit-h and parakeet conformer forward passes. so i wrote that runtime in pure mlx. vision tower, audio tower, the processor and the token splicing, all ported from nvidias reference implementation. it runs the mlx-community 4bit quant for the language model with the two towers in bf16. i didnt want to guess whether it was actually right, so every component gets tested against nvidias pytorch reference on the same inputs with the same weights. 23 of 23 passing. audio tower cosine 0.99999130 min per frame, vision tower 0.99996227 min per token, and on the mlx cpu stream the vision tower comes out graph exact at 1.0. on my m5 max it does 67.7 tok/s with an image, 147 tok/s with audio, 152 tok/s text only. wifi off the whole time. hand it a screenshot and it reads it, hand it an audio clip and it hears it. its mine and its mit licensed. https://github.com/nicedreamzapp/nemotron-omni-mlx peak was 22.1gb on the image path so it should fit on a 32gb mac, but the m5 max is the only thing i have to test on. if you run it on something smaller id like to hear what happens. credit to nvidia for the open weights and to yayr for the 4bit conversion. submitted by /u/divinetribe1 [link] [comments]

Related

• Nemotron-3-Nano-Omni-30B-A3B-Reasoning, New model?
• NVIDIA Nemotron 3 Nano Omni Powers Multimodal Agent Reasoning in a Single Efficient Open Model
• Nemotron 3 Nano Omni: Efficient and Open Multimodal Intelligence

Source: r/LocalLLaMA | 2026-08-06