VDN-H3 on M5 Pro 24GB: up to ~2.6× faster H3 generation

I recently ported VDN-H3 (Video DeltaNet for MiniMax H3) to Vpipe and tested it on an M5 Pro 24GB. The benefit scales with both sequence length and resolution: at 832×480, VDN-H3 starts to pull ahead at around 6s and reaches ~1.7× at 15s; at 1344×768, the crossover happens much earlier and the speedup reaches ~2.6× at ~14s. All tests use 6 DiT steps with Turbo LoRA. VDN-H3 project Vpipe is a native C++20/Metal inference runtime for Apple Silicon, without PyTorch/MPS/MLX. Most of the underlying infrastructure was already in place, so porting VDN-H3 was fairly mechanical and took about one day end-to-end. Vpipe project https://preview.redd.it/3bxtes3571oh1.png?width=2673&format=png&auto=webp&s=bc6292466f7f224ae8f1ff70feff803574bb8e89 https://preview.redd.it/bvf33s3571oh1.png?width=2673&format=png&auto=webp&s=7a04f7b04267671f5e534ba4bb251fb90274447d Another recent effort in this space is FastH3, which focuses on accelerating H3 for local inference and recently released an MLX implementation for Apple Silicon. They reported 465–504s on an M4 Max 36GB for 832×480, 124 frames and 4 steps. FastH3 local implementation & benchmarks As a rough reference point, Vpipe's baseline H3 without VDN takes 469s on a 16GB M5 MacBook Air for the same resolution and frame count, while running 6 DiT steps with Turbo LoRA. This was a full cold start, including the text encoder, with no model weights cached. It's not a controlled hardware comparison, but I found it interesting that baseline H3 on a 16GB Air lands in essentially the same latency range. I like the core idea behind VDN: exploiting the inherent redundancy in video diffusion rather than applying the same amount of computation uniformly across the sequence. The scaling behavior above seems to support that direction, especially for larger and longer generations. VDN-H3 uses a hybrid architecture combining a frame-wise linear-attention branch with a softmax-attention branch. There is a memory tradeoff. VDN-H3 adds roughly 4GB of weights that cannot be fused into the base model, plus another ~1GB that can potentially be fused LoRA-style. Vpipe doesn't fuse the latter yet. On a 24GB system, this extra memory pressure can outweigh the compute savings for smaller workloads. There should also be room to reduce the overhead by 8-bit quantizing the incremental weights. Quality-wise, with the same seed, VDN-H3 generally preserves a composition very similar to the baseline H3 output, which I think is a nice property. There is visible degradation in some sequences, though. In one test with an upward-moving column of water, the flow reversed downward for a few seconds midway through the video, then switched back upward near the end. I suspect this kind of long-range temporal inconsistency may be related to the heavy use of sliding-window and linear attention. Mixing in occasional global attention might be worth exploring. Reference conditioning is another interesting open area, particularly first/last-frame and generic reference inputs, since these are more challenging to handle cleanly under the sliding-window/linear-attention structure. submitted by /u/TgoAI [link] [comments]

Source: r/StableDiffusion | 2026-09-07

Related