Search: “benchmark”

arXiv:2609.30722v1 Announce Type: cross Abstract: Existing roadside traffic datasets support perception, forecasting, and visual question answering, but they do not evaluate counterfactual video gener

arXiv:2609.30055v1 Announce Type: cross Abstract: In the Era by Eon benchmark, each question states the rules for its answer, and code computes the answer from a generated company's data. When agents

Would be interesting right? Models are starting to score higher and higher on benchmarks like Parametric CAD Bench, but can they pass an actual exam? The Certified SOLIDWORKS Associate (CSWA) exam mig

arXiv:2604.01754v2 Announce Type: replace-cross Abstract: Mathematical reasoning is a hallmark of human intelligence, and whether large language models (LLMs) can meaningfully perform it remains a cen

arXiv:2609.28952v1 Announce Type: new Abstract: Robot-policy benchmarks increasingly cover diverse tasks and preset out-of-distribution conditions, but typically evaluate complete trajectories from pr

arXiv:2609.30027v1 Announce Type: new Abstract: Frontier language models are rarely used in clinical workflows because the realistic, longitudinal benchmarks needed to develop them are scarce. Real el

arXiv:2609.29740v1 Announce Type: cross Abstract: Ligand-based virtual screening (LBVS) is a practical first-pass tool in early-stage drug discovery, but existing benchmarks can overestimate performan

In September 2026, Sebastian Thrun announced **PhilosophyBench**, the first independent, large‑scale benchmark created by Stanford’s AI Lab and HCI group to evaluate artificial intelligence’s philosop

arXiv:2609.27176v1 Announce Type: new Abstract: Evidence that evaluation material entered training does not reveal how much it affected evaluation. This distinction leaves a contaminated benchmark sco

Today, @FactoryAI joins the Specialized Intelligence Index with Legacy-Bench! The SII keeps growing, now with benchmarks testing how models handle the legacy software behind payroll, financial settlem

arXiv:2609.22878v1 Announce Type: new Abstract: Large language model code generation benchmarks primarily evaluate well-resourced languages like Python and Java, where models benefit from abundant tra

I’m the developer of TensorSharp. Inspired by vLLM PR #57250, I added and optimized a native /v1/systemone endpoint for Jev-style structured decisions using DiffusionGemma GGUF. This uses DiffusionGem

We benchmarked Jev against Claude Opus 5 and GPT-5.6 Terra on accuracy, cost, and latency. Learn how threshold tuning changes hallucination detection. The post Jev vs. LLM-as-a-Judge: Accuracy and cos

Standardizing independent evaluation, fairness, and interpretability for mechanism-informed generative AI in healthcare The post MLCommons Joins EU-Funded AIRIS Project to Build and Benchmark Next-Gen

arXiv:2609.25689v1 Announce Type: new Abstract: Recent advances in learning-based robot policies have demonstrated promising progress, yet they are predom- inantly evaluated in static or quasi-static

arXiv:2609.22586v1 Announce Type: cross Abstract: Modern audio-language models are no longer judged only on what words they can transcribe, but on whether they can reason over what they hear: recoveri

arXiv:2602.13217v2 Announce Type: replace Abstract: Reasoning benchmarks need renewal along two axes: freshness and headroom. VeRA makes both executable and auditable by turning each item into a task

arXiv:2609.22183v1 Announce Type: new Abstract: Public benchmark scores may reflect skill, prior exposure to the questions, or both, and for most models the training data are unknown. We present Clean

arXiv:2609.24350v1 Announce Type: cross Abstract: Robotic foundation models achieve impressive performance on standard manipulation benchmarks, yet these evaluations typically assume clean, timely, an

MiMo-V2.6 is 'simply' the best (for now). Despite its simple architecture design it's currently No.1 in the open-weight benchmarks (weighted average). With 'simple,' I mean a classic Grouped Query Att

arXiv:2609.22591v1 Announce Type: new Abstract: Robot learning policies fail in characteristic ways: they stall in uncertain states, drift during contact-rich alignment, and miss targets by millimetre

arXiv:2606.16560v2 Announce Type: replace Abstract: Automatic semantic change detection aims to identify how word meanings shift over time, offering insights into both linguistic and societal change.

arXiv:2609.24444v1 Announce Type: new Abstract: Accurate, reliable, and deployable wind power forecasting is critical for power system dispatch, renewable energy integration, and electricity market op

arXiv:2609.21386v1 Announce Type: cross Abstract: Comprehensive video understanding is crucial for advancing artificial intelligence toward the intricate dynamics of the physical world. While recent a

arXiv:2609.21879v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) have recently been proposed as promising tools for face recognition, as they can produce natural language explanations a

arXiv:2603.13099v3 Announce Type: replace Abstract: We introduce CRYSTAL (Clear Reasoning via Yielded Steps, Traceability, and Logic), a diagnostic benchmark with 6,372 instances that evaluates multim

arXiv:2609.21267v1 Announce Type: new Abstract: Production LLM agents are evaluated repeatedly as they evolve, but full agent benchmarks are costly to rerun. We study efficient recurring evaluation fo

Grok 4.7 xHigh is now at the top with just ONE point away from Claude Fable 5.1 Max on Artificial Analysis’ AA-Briefcase benchmark Claude Fable 5.1 Max — 59% Grok 4.7 xHigh — 58% Just a 1-point differ

Carl Franzen / VentureBeat: MiMo-V2.6-Pro ties Grok 4.7 (xHigh) and beats GLM-5.3 (max) on Artificial Analysis' Intelligence Index, making it the benchmark's top-scoring open-weight model — That puts

Xiaomi: Xiaomi debuts open-weight omnimodal models MiMo-V2.6 Pro and Flash; Pro allegedly performs “on par with Opus 5 and GPT-5.6 Sol across most agent benchmarks” — Frontier intelligence, all the mo

Live now: our Inference Engineering Track from AI Engineer World's Fair 2026. A benchmark tool told to run 200 queries a second that ran 38 and reported 200. A model that answered one prompt in a thou

arXiv:2609.19840v1 Announce Type: new Abstract: High-quality 3D talking face datasets remain largely English- centric, and Korean 3D facial motion data are difficult to combine with standard English b

arXiv:2609.19883v1 Announce Type: cross Abstract: Characterizing LLM reasoning remains an open challenge, as many existing benchmarks isolate specific reasoning skills, rely on external knowledge, or

arXiv:2609.19182v1 Announce Type: new Abstract: Benchmarks are central to how progress in large language models (LLMs) is assessed and communicated. Yet model rankings alone reveal little about how ev

just saw some benchmarks on the omlx website for the m5 ultra (don’t know how official they are but they seem reasonable): Link For Qwen 3.8 27B q4 it gets 50 tok/s th and 1800 tok/s pp at8k context a

arXiv:2609.18680v1 Announce Type: new Abstract: Contextual ASR can benefit from semantic cues or from target words explicitly provided in the context. We introduce HearInContext, a Mandarin--English b

arXiv:2609.16739v1 Announce Type: new Abstract: Background: Large language models (LLMs) have achieved physician-comparable performance on multiple-choice medical knowledge examinations, but their cap

arXiv:2609.16012v1 Announce Type: new Abstract: Despite significant progress in general visual question answering and cross-modal understanding, multimodal large language models still face a pronounce

arXiv:2609.15726v1 Announce Type: cross Abstract: Tactile sensing provides contact information that can be difficult to infer from vision alone, but tactile hardware for dexterous hands has not conver

arXiv:2609.14654v1 Announce Type: new Abstract: Convolutional neural networks (CNNs) are typically evaluated using held-out classification accuracy, an approach that presupposes predictions are based

arXiv:2609.14676v1 Announce Type: new Abstract: Purpose: To compare four multimodal large language models (MLLMs) with radiologists of varying expertise in breast density assessment, BI-RADS assessmen

arXiv:2609.13606v1 Announce Type: cross Abstract: Multi-arm robotic harvesting offers a promising path to improve harvesting efficiency and reduce reliance on manual labor. However, practical deployme

arXiv:2609.12808v2 Announce Type: replace Abstract: Unlearning benchmarks such as TOFU and MUSE certify forgetting by reading the model's final answer, where a model that refuses to answer already cou

arXiv:2609.15794v1 Announce Type: new Abstract: Clinical AI systems are evaluated with instruments built for research settings (reference-based similarity metrics and expert rubric panels) that measur

arXiv:2609.14860v1 Announce Type: cross Abstract: Warning: This submission studies stereotypes and biases, and contains toxic and offensive examples, used for illustration purposes only. Fairness benc

Valida Pau / The Information: Source: AI assistant Instinct seeks to raise 1B at a 10B valuation, up from 2.25B less than a month ago; Sequoia and Benchmark are in talks to lead the round — The startu