Model Releases
QontoFAQ: A better Information Retrieval Benchmark [R]
Retrieval benchmarks sometimes feel benchmaxxed by models, so we wanted to find a way to tie it as close as possible to my objective: finding the article that answers a product question right. We work
Retrieval benchmarks sometimes feel benchmaxxed by models, so we wanted to find a way to tie it as close as possible to my objective: finding the article that answers a product question right. We worked on a new metric which seems more proportional to document relevance, and built up a benchmarking dataset to measure embedding models. Here is an article on the approach: https://medium.com/qonto-way/qontofaq-benchmarking-information-retrieval-acd89600ebe1 and the associated code: https://github.com/qonto/qonto-faq-benchmark submitted by /u/espadrine [link] [comments]
Related
- Open-source 9-task benchmark for coding-agent retrieval augmentation. Per-task deltas +0.010 to +0.320, all evals reproducible [P]
- Putting the question before the context took my local Qwen from 89% to 100% on a decision benchmark, and from ~400 ms to ~80 ms
- AQUA-Bench: Beyond Finding Answers to Knowing When There Are None in Audio Question Answering
- Measuring LLM performance drift: observations and methodology from 31,352 repeated benchmark measurements [D]
Source: r/MachineLearning | 2026-09-22