Entropy-Coded MS-VQ-VAE with Learned Priors for Ultra-Low Bitrate Video Compression

arXiv:2607.02562v1 Announce Type: new Abstract: Learned video codecs based on continuous latent representations struggle to operate reliably below 0.1 bits per pixel~(bpp): without a differentiable rate signal, Lagrangian optimisation cannot effectively trade reconstruction quality for bitrate at extreme compression ratios. We demonstrate that discrete latent representations sidestep this limitation entirely. In a vector-quantized~(VQ) codec, the codebook sizeK imposes a hard information ceiling of log_2 K bits per symbol; a learned autoregressive prior then exploits the non-uniform distribution of code usage -- which we show follows a power law -- to push actual bitrates well below this ceiling, without any rate-penalty tuning. Building on the MS-VQ-VAE architecture introduced inite{kotthapalli2026msvqvae}, we sweep K in {128, 256, 512, 1024} under a uniform training protocol to trace four operating points on the rate-distortion~(RD) curve. We identify and resolve a critical training instability: gradient-based VQ collapses catastrophically at K leq 512, whereas EMA-stabilised codebook updates with dead-code restart maintain full utilisation across all configurations. On 500 UCF101 test clips (64!imes!64, 32frames), our models operate at 0.043-0.064bpp -- 3.3-5imes below H.264's practical floor and 5-7.6imes below H.265's floor at this resolution. Every MS-VQ-VAE configuration outperforms H.265 CRF,36 on perceptual quality (LPIPS) despite using 5-7.6imes fewer bits. At K{=}1024, the model surpasses H.265 CRF,36 on LPIPS by a margin of 0.072 absolute while using 5.1imes fewer bits. Codebook analysis confirms power-law index distributions and 70-85% entropy efficiency, establishing the pipeline as a principled learned entropy coder.

Source: arXiv cs.CV | 2026-07-07