Search: “benchmark”

The FFASR Leaderboard is a benchmarking platform introduced by Hugging Face for evaluating Automatic Speech Recognition (ASR) systems on real-world performance metrics. It provides a standardized eval

arXiv:2606.24200v1 Announce Type: cross Abstract: Retrieval-augmented generation (RAG) in clinical settings increasingly requires multilingual retrieval against predominantly English evidence corpora.

arXiv:2603.16131v2 Announce Type: replace Abstract: The explosive growth of AI research has created unprecedented information overload, increasing the demand for scientific summarization at multiple l

We benchmarked Mistral OCR against other frontier and open-weight models on ParseBench 📊 For a model at its price point, it is quite competitive! - It wins on semantic formatting - understanding strik

arXiv:2606.22338v1 Announce Type: cross Abstract: Robots deployed in realistic settings will accumulate experience across many sessions and tasks over their deployment. The robot's tasks may often req

arXiv:2606.21949v1 Announce Type: new Abstract: Accurate and comprehensive video captions with consistent subject references are critical for downstream understanding and generation tasks. However, fe

arXiv:2606.20961v1 Announce Type: new Abstract: Continual adaptation is essential for multimodal large language models (MLLMs) deployed across evolving domains, but the state-of-the-art MR-LoRA method

arXiv:2505.17295v2 Announce Type: replace Abstract: We introduce ScanBot, a benchmark for instruction-conditioned, high-precision surface scanning with robot-mounted industrial laser profilers. Unlike

arXiv:2603.06608v2 Announce Type: replace-cross Abstract: The research community lacks a middle ground between StarCraft II full game and its mini-games. The full-game's sprawling state-action space r

Benchmarks tell only part of the story. Fugu’s real value shows up in long, messy, real-world workflows. During our beta with 500 users, we saw Fugu Ultra drive meaningful progress in fully automated

GLM-5.2 leads open weights models and sits at #3 overall on GDPval-AA, a real-world agentic work benchmark GLM-5.2 from @Zai_org scores 1524 Elo on GDPval-AA, which measures performance on real-world,

Pamela Fox created a custom t-shirt design referencing Simon Willison's Pelican benchmark, a technical tool or metric in web development or performance testing. She posted about it on X (formerly Twit

Carl Franzen / VentureBeat: Sakana AI launches Fugu, a multi-agent orchestration system accessible through a single model API, claiming Fugu Ultra matches Fable and Mythos on benchmarks — Last night,

We've kept hearing how GLM-5.2 beats Opus 4.8, and are skeptical of benchmarks - so we tested them on a real bug from the Cline repo. While both models fixed the issue, GLM was the winner in terms of

arXiv:2405.06995v4 Announce Type: replace-cross Abstract: Automated deception detection is crucial for assisting humans in accurately assessing truthfulness and identifying deceptive behavior. Convent

arXiv:2601.04203v2 Announce Type: replace Abstract: We present FronTalk, a benchmark for front-end code generation that pioneers the study of a unique interaction dynamic: conversational code generati

arXiv:2606.11702v1 Announce Type: cross Abstract: To make clinically grounded decisions, medical AI agents are expected to go beyond simple recognition and be capable of tool retrieval, evidence acqui

arXiv:2606.10066v1 Announce Type: cross Abstract: Medical vision-language models (VLMs) are evaluated on public benchmarks whose images and question-answer pairs have been freely downloadable for year

Sarah Guo: As AI commoditizes benchmarkable work, an organization's lasting moats lie in tasks that are verifiable through its private data and judgment — The mid-2026 investor's version of AI psychos

arXiv:2606.10554v1 Announce Type: cross Abstract: Large Language Models (LLMs) are increasingly deployed in real-world applications that require access to up-to-date knowledge. However, retraining LLM

arXiv:2606.10061v1 Announce Type: new Abstract: Large language models (LLMs) increasingly participate in emotionally sensitive social conversations, where responses may shift from balanced support tow

arXiv:2606.10673v1 Announce Type: cross Abstract: Although some very common test beds exist for assessing the performance of clustering methods, large scale benchmarking is typically limited to relati

Cohere has released Transcribe, an open-source speech recognition model that achieved the top ranking on Hugging Face's newly established Far-Field Automatic Speech Recognition (ASR) benchmark. The mo

arXiv:2606.10852v1 Announce Type: cross Abstract: LLM deception is often evaluated through direct markers such as fabricated claims, explicit lies, or strategic concealment. However, many real-world m

arXiv:2606.11144v1 Announce Type: new Abstract: Resistance to first-line osimertinib in EGFR-mutant non-small-cell lung cancer (NSCLC) is the canonical example of predictable clonal evolution under th

arXiv:2606.11152v1 Announce Type: new Abstract: Multimodal large language models can write code to produce complex programs as well as use programs to do 3D modeling, which opens up a new avenue for 3

arXiv:2606.10255v1 Announce Type: cross Abstract: Cryo-electron tomography (cryoET) has emerged as a powerful tool in structural and cellular biology by enabling direct visualization of macromolecular

arXiv:2606.10394v1 Announce Type: new Abstract: Large language models are increasingly used to power personal agents for everyday applications, but evaluating these agents remains a challenge. Existin

arXiv:2606.11070v1 Announce Type: cross Abstract: Recent advances in reasoning and tool-calling capabilities of large language models (LLMs) have enabled increasingly capable agentic systems. However,

arXiv:2606.10382v1 Announce Type: new Abstract: Real-robot evaluation is essential for understanding whether learned manipulation policies can operate reliably outside curated demonstrations. This nee

Want to learn more about the Far-Field ASR benchmark? Join Treble's webinar tomorrow, June 11th, with @shinjiw_at_cmu, Cohere's @Julianfmack, and other industry leaders discussing the future of far-fi

FrontierCode is a benchmarking framework designed to evaluate code quality and move beyond 'slop'—low-quality or poorly generated code—in AI-generated outputs. The framework likely provides metrics an

This paper benchmarks frontier automatic speech recognition (ASR) systems on their ability to handle code-switched speech, where bilingual speakers mix languages within a single conversation. The rese

arXiv:2603.13431v3 Announce Type: replace-cross Abstract: Computational antibody design has seen rapid methodological progress, with dozens of deep generative methods proposed in the past three years,

Claude Fable 5 has been integrated into Devin and achieved the top ranking on FrontierCode, Cognition AI's benchmark for real-world engineering tasks. This integration represents an advancement in Dev

arXiv:2606.07542v1 Announce Type: cross Abstract: Generative AI is reshaping healthcare, yet most existing advances rely on hospital-grade devices, which limits their accessibility and potential for h

arXiv:2606.09276v1 Announce Type: new Abstract: Equation discovery aims to automate the discovery of scientific models in the form of mathematical equations from data. Technically, equation discovery

arXiv:2606.08594v1 Announce Type: new Abstract: Deep learning EEG denoising architectures have scaled from tens of thousands to tens of millions of parameters, yet no prior study has isolated model ca

A developer created a Windows GUI tool for managing and benchmarking different llama.cpp builds, addressing the needs of AMD GPU users who work with multiple compute backends like Vulkan, ROCm, and HI

arXiv:2606.09764v1 Announce Type: new Abstract: A useful phone agent needs to be personally intelligent. It should reason over a user's identity, history, and preferences as they exist on the device,

arXiv:2606.08729v1 Announce Type: cross Abstract: Simulation plays a key role in automated robotics research supported by large language models (LLMs). However, existing simulators often require custo

arXiv:2606.07550v1 Announce Type: cross Abstract: Offline reinforcement learning (RL) offers a promising route for developing plasma controllers from historical tokamak data, since online trial-and-er

arXiv:2606.08572v1 Announce Type: new Abstract: While Omni-modal Large Language Models (OLLMs) have demonstrated impressive capabilities in jointly processing audio and visual streams, their ability t

Small: 30 billion parameters, 3B active. Efficient: Benchmarks to 33.4 on the Artificial Analysis Coding Index, competitive among similar sized models. Open Source: Apache 2.0 license so developers ca

arXiv:2606.07929v1 Announce Type: new Abstract: Large language models (LLMs) are entering clinical practice based on benchmark accuracy that may fail to detect safety-relevant failure modes. Here we p

arXiv:2606.07953v1 Announce Type: new Abstract: Large-scale Visual-Language Models (LVLMs) have achieved remarkable success in natural visual tasks, yet their application to industrial defect detectio