Codebook Capacity Governs Perceptual Quality Across Resolutions in Hierarchical Discrete Video Compression

arXiv:2607.23366v1 Announce Type: cross Abstract: Learned video codecs based on continuous latent representations typically require resolution-specific retraining or rate-distortion (RD) recalibration when scaling to new spatial resolutions, because entropy models and Lagrangian weights are tightly coupled to the operating point. We investigate whether hierarchical discrete latent codecs exhibit the same sensitivity. Using a controlled empirical study of MS-VQ-VAE video compression across codebook sizes K in {128,256,512,1024} and resolutions 64imes64, 128imes128, and 256imes256 on UCF101, we show that perceptual quality (LPIPS) depends strongly on codebook capacity but only negligibly on spatial resolution. Fitting a log-linear model Q(K,r) = alphalog_2 K + etalog_2 r + gamma to all 12 operating points yields alpha=-0.0094 (t=-6.6, p<0.001) and eta=-0.0009 (t=-0.43, p=0.68, not significant), with R^2=0.82. Codebook capacity is therefore roughly 10imes more influential than spatial resolution per log-unit increase. In parallel, bottom-level entropy efficiency eta=H(z)/log_2 K remains stable or improves with resolution (84-87% at 64imes64; 92-94% at 256imes256), confirming that larger spatial grids are utilized more efficiently rather than less. Across all resolutions and codebook sizes, our models outperform H.264 on LPIPS at matched or lower bitrate, with gains of 25-52% at 128imes128 and 21-37% over H.265 at 256imes256. These findings suggest that codebook size K, not spatial resolution, is the dominant design variable governing perceptual compression quality in hierarchical discrete video codecs -- a property that may simplify multi-resolution deployment and inform the design of scalable discrete tokenizers for generative video models.

Source: arXiv cs.CV | 2026-07-28

Related