Beyond Retrieval: A Multitask Benchmark and Model for Code Search

arXiv:2605.04615v2 Announce Type: replace-cross Abstract: Code search has usually been evaluated as first-stage retrieval, even though production systems rely on broader pipelines with reranking and developer-style queries. Existing benchmarks also suffer from data contamination, label noise, and degenerate binary relevance. In this paper, we introduce extsc{CoREB}, a contamination-limited, multitask nderline{co}de nderline{r}etrieval and rnderline{e}ranking nderline{b}enchmark, together with a fine-tuned code reranker, that goes beyond retrieval to cover the full code search pipeline. extsc{CoREB} is built from counterfactually rewritten LiveCodeBench problems in five programming languages and delivered as timed releases with graded relevance judgments. We benchmark eleven embedding models and five rerankers across three tasks: text-to-code, code-to-text, and code-to-code. Our experiments reveal that: ircone code-specialised embeddings dominate code-to-code retrieval ({sim}2{imes} over general encoders), yet no single model wins all three tasks; irctwo short keyword queries, the format closest to real developer search, collapse every model to near-zero nDCG@10; ircthree off-the-shelf rerankers are task-asymmetric, with a 12-point swing on code-to-code and no baseline net-positive across all tasks; ircfour our fine-tuned extsc{CoREB-Reranker} is the first to achieve consistent gains across all three tasks. The data and model are released.

Source: arXiv cs.AI | 2026-05-11