When do cheap embeddings beat protein language models? A theoretically-grounded hashing sketch for biological sequence classification

arXiv:2512.10147v2 Announce Type: replace Abstract: extbf{Motivation:} Pre-trained protein language models (PLMs) such as ESM-2 have become the default representation for biological sequence tasks, but they are computationally heavy and require GPUs both for embedding and for fine-tuning. Whether they are actually necessary for sequence classification, as opposed to structure prediction, is rarely tested against strong, principled, lightweight alternatives. This question has direct practical stakes for large-scale genomic surveillance, where embedding millions of sequences on commodity hardware is a recurring bottleneck. extbf{Results:} We introduce Murmur2Vec, an alignment-free, training-free embedding that aggregates k-mer counts into a small hash table via the deterministic MurmurHash function, and we cast it as a randomized sketch of the classical k-mer spectrum kernel. We provide a complete theoretical treatment: closed-form bias/variance of the inner product, an unbiased signed variant with a Johnson--Lindenstrauss-type concentration bound, an excess-risk bound for downstream linear classifiers that makes the bias--variance trade-off in the hash-table size explicit, and an implicit-regularization mechanism by which collisions damage frequent non-discriminative k-mers more than rare lineage-defining ones. Across four classification tasks, SARS-CoV-2 spike lineage (22 classes), HIV-1 Env subtype (8 classes), and two protein-family benchmarks (8 and 6 classes), Murmur2Vec matches a LoRA-fine-tuned 650M-parameter ESM-2 model on the two tasks for which LoRA fine-tuning was run to convergence (SARS-CoV-2 and HIV-1) and ties frozen ESM-2 on the two protein-family tasks, and it outperforms the fine-tuned model on the hardest task (SARS-CoV-2 lineage: 0.854 vs. 0.807 accuracy; macro-F1 0.684 vs. 0.401).

Source: arXiv cs.LG | 2026-09-11

Related