Research
[P] PCA before truncation makes non-Matryoshka embeddings compressible: results on BGE-M3 [P]
Applying PCA as a rotation step before naively truncating embeddings from non-Matryoshka models like BGE-M3 can recover much of the retrieval quality that is otherwise lost when simply chopping dim...
Applying PCA as a rotation step before naively truncating embeddings from non-Matryoshka models like BGE-M3 can recover much of the retrieval quality that is otherwise lost when simply chopping dimensions, because PCA reorganizes the embedding space to maximize variance explained, concentrating the most informative signal into the leading dimensions . BGE-M3 does not natively support Matryoshka representation, meaning changing output dimensions without any post-processing leads to poor results . By fitting a PCA transform on a representative corpus and applying it prior to truncation, practitioners can make standard fixed-dimension models like BGE-M3 behave similarly to Matryoshka-trained models, enabling meaningful dimensionality reduction without retraining.
Related
- What if your HNSW index stored 3-bit embeddings instead of float32? [R]
- Entropy-Gradient Grounding: Training-Free Evidence Retrieval in Vision-Language Models
- What if your HNSW index stored 3-bit embeddings instead of float32? [R]
- [[d-large-scale-ocr-d|[D] Large scale OCR [D]]]
Source: research