Geometric Properties of the Voronoi Tessellation in Latent Semantic Manifolds of Large Language Models

arXiv:2604.06767v1 Announce Type: new Abstract: Language models operate on discrete tokens but compute in continuous vector spaces, inducing a Voronoi tessellation over the representation manifold. We study this tessellation empirically on Qwen3.5-4B-Base, making two contributions. First, using float32 margin recomputation to resolve bfloat16 quantization artifacts, we validate Mabrok's (2026) linear scaling law of the expressibility gap with R^2 = 0.9997 - the strongest confirmation to date - and identify a mid-layer geometric ambiguity regime where margin geometry is anti-correlated with cross-entropy (layers 24-28, rho = -0.29) before crystallizing into alignment at the final layer (rho = 0.836). Second, we show that the Voronoi tessellation of a converged model is reshapable through margin refinement procedures (MRP): short post-hoc optimization runs that widen token-decision margins without retraining. We compare direct margin maximization against Fisher information distance maximization across a dose-response sweep. Both methods find the same ceiling of ~16,300 correctable positions per 256K evaluated, but differ critically in collateral damage. Margin maximization damage escalates with intervention strength until corrections are overwhelmed. Fisher damage remains constant at ~5,300 positions across the validated range (lambda = 0.15-0.6), achieving +28% median margin improvement at lambda = 0.6 with invariant downstream benchmarks - a geometric reorganization that compresses the expressibility gap while preserving its scaling law. However, frequency and token-class audits reveal that gains concentrate in high-frequency structural tokens (84% of net corrections at lambda = 0.6), with content and entity-like contributions shrinking at higher lambda. Fisher MRP is therefore a viable geometric polishing tool whose practical ceiling is set not by aggregate damage but by the uniformity of token-level benefit.

Related

• Improving Semantic Uncertainty Quantification in Language Model Question-Answering via Token-Level Temperature Scaling
• What Drives Representation Steering? A Mechanistic Case Study on Steering Refusal
• Are Face Embeddings Compatible Across Deep Neural Network Models?
• Reasoning Within the Mind: Dynamic Multimodal Interleaving in Latent Space

Source: arXiv cs.LG | 2026-04-10