Search: “benchmark”

arXiv:2610.21654v1 Announce Type: new Abstract: This paper presents the Digital Typhoon Dataset V3, a new version of the longest typhoon satellite image dataset spanning nearly 50 years, designed to b

arXiv:2610.21412v1 Announce Type: new Abstract: Large language model (LLM) agents now fix software, operate computers and serve customers, but their harnesses, the programs that run them, call a large

Mistral AI announced in mid‑October 2026 that its new model, Mistral Large 4, achieved high scores on Ulam’s private STEM benchmark (STEMBench), particularly excelling in engineering and astronomy cat

arXiv:2610.21659v1 Announce Type: cross Abstract: Corporate financial disclosures often unfold over time, with announcements, prepared remarks, and discussion spanning multiple high-frequency return i

arXiv:2610.14700v1 Announce Type: new Abstract: Most text-to-SQL benchmarks provide the target database. In a large data catalog, however, choosing the sources is part of the problem: an agent must fi

arXiv:2610.19564v1 Announce Type: new Abstract: A spatial answer depends on the reference frame specified in language and its supporting visual evidence. Item accuracy alone obscures whether correctne

Hermes Agent is the best place to use the new @NVIDIAAI Nemotron 3.5 Super VL and Bolt models. NVIDIA benchmarked Super VL on Hermes Bench with a score of 63.44, and compared to similar local models i

I calculated KLD and perplexity for Qwen3.8 27B models that fit 16GB VRAM: See comments for the benchmarks. Reddit filters keeps deleting the post. Conclusion: q5_1/q5_1 is almost always better than q

arXiv:2610.19159v1 Announce Type: cross Abstract: Can AI agents carry out research on open scientific problems? We introduce Into the Unknown, a benchmark that provides an agent with a science paper a

Jeffrey Kampman / Tom's Hardware: Nvidia's RTX Spark Superchip in the Surface Laptop Ultra outperforms Apple's M5 Max MacBook Pro by ~25% in agentic AI completion benchmarks using Qwen models — Laying

arXiv:2601.08689v3 Announce Type: replace Abstract: Large Language Models (LLMs) have shown strong capabilities in finance-oriented question answering, yet their evaluation on professional quantitativ

arXiv:2610.16942v1 Announce Type: cross Abstract: Large audio-language models (LALMs) are fast becoming a general interface to sound, yet how far their broad pretraining carries into specialized domai

arXiv:2610.15179v1 Announce Type: cross Abstract: Large language models are increasingly deployed in contexts requiring adherence to ethical and legal norms, yet their alignment with internationally r

arXiv:2610.16577v1 Announce Type: new Abstract: Reconstructing reference videos as editable dynamic 3D scenes is an important path toward controllable video generation for filmmaking: these scenes pro

arXiv:2610.15345v1 Announce Type: new Abstract: High-quality coding problems are expensive to collect. Benchmarks such as SWE-bench rely on substantial human curation, while synthetic alternatives typ

arXiv:2610.17794v1 Announce Type: new Abstract: Contextual forecasting seeks to improve time series forecasting by incorporating forward-looking information, such as planned events, external condition

arXiv:2610.18843v1 Announce Type: new Abstract: Robot policies are expected to generalize across deployment environments, including variations in the robot itself: cameras are bumped and remounted, ca

arXiv:2602.05382v2 Announce Type: replace Abstract: Vision-language model (VLM) errors on visual reasoning may come from missing visual structure, from mistakes after that structure is extracted, or f

arXiv:2610.12669v1 Announce Type: cross Abstract: Large language models are becoming central to GeoAI systems, yet their evaluation rarely considers the elicitation of geospatial data requirements. Th

🇫🇷 Le Chonk joins the sudo L7 leaderboard 🇫🇷 We just added @MistralAI's Mistral Large 4 to sudo L7, our benchmark for staff-level software engineering. Mistral Large 4 scores 13.3%, ahead of Muse Spar

arXiv:2610.12561v1 Announce Type: new Abstract: Memory allows large language model (LLM) agents to use past experience, often stored as textual records, when responding to requests. For embodied agent

arXiv:2610.11065v1 Announce Type: new Abstract: Automating city-scale wireless deployment remains challenging under complex urban propagation and network-wide interference. We introduce extbf{CityDepl

arXiv:2610.11694v1 Announce Type: cross Abstract: Existing reaction models primarily learn molecular transformations, whereas enzy- matic reactions depend jointly on molecular structure and catalytic

arXiv:2610.08966v2 Announce Type: replace Abstract: Humans perceive far more in a scene than what is explicitly depicted: a single glance captures past causes and future trajectories; a quick peek det

arXiv:2610.12089v1 Announce Type: new Abstract: Long-horizon mobile manipulation requires a robot to navigate multi-room environments and execute a sequence of manipulation skills under a single natur

arXiv:2602.00811v2 Announce Type: replace Abstract: Current Multimodal Affective Computing (MAC) systems heavily rely on the completeness of multiple modalities to accurately understand human's affect

arXiv:2610.12409v1 Announce Type: new Abstract: Safety benchmarks typically report one overall score for a suite of datasets, each of which may target one or more safety-related attributes, so models

All anyone ever talks in this sub is about x new model having x benchmark scores and how well it can code. Or some new tech to increase t/s for qwen. So I wanted to go back to the roots of this sub an

arXiv:2610.10047v1 Announce Type: new Abstract: Product-centric advertisement video generation aims to create promotional videos that preserve fine-grained product identity while presenting selling po

I'm developing a benchmark for a low resource language and I don't want it to be leaked and used for training when it is being used to get predictions. For locally run models it shouldn't be a problem

arXiv:2510.01335v2 Announce Type: replace Abstract: The manifold hypothesis suggests that data lies on manifolds with smaller intrinsic dimension (ID) than their ambient dimension. However there is no

arXiv:2610.10125v1 Announce Type: cross Abstract: Household robots need to continually navigate to different objects in the same environment, many of which are small and portable, such as tools and to

arXiv:2610.06988v1 Announce Type: new Abstract: Knowledge tracing (KT) models are predominantly evaluated using aggregate metrics such as area under the curve (AUC) and accuracy. However, these global

arXiv:2610.07232v1 Announce Type: new Abstract: Accurate short-term load forecasting (STLF) is essential for the reliable and efficient operation of modern power systems. While time series foundation

arXiv:2610.07616v1 Announce Type: new Abstract: We present a repertoire-free benchmark for soft-robot damage recovery under nine online trials. The protocol pairs damage masks within each morphology,

arXiv:2610.07026v1 Announce Type: new Abstract: The Offline AI Modules workstream enables practical, low-power, and community-accessible deployment of voice-first AI systems that operate fully offline

arXiv:2610.08684v1 Announce Type: new Abstract: Modern video models can generate realistic videos from real appearance references and proxy renders that specify scene structure, viewpoint changes, and

arXiv:2610.08161v1 Announce Type: cross Abstract: Ambient documentation systems are rapidly gaining adoption, yet their impact on clinical note quality remains poorly characterized. We introduce MedCo

arXiv:2610.08733v1 Announce Type: new Abstract: Language models provide a plug-and-play interface between humans and robots, but important challenges remain when speech, dialogue, fast interaction, an

arXiv:2610.07802v1 Announce Type: new Abstract: Bird monitoring in natural environments is challenging due to the small size of some species of birds relative to the scene, background clutter, variabi

arXiv:2406.12150v2 Announce Type: replace Abstract: Post-hoc attribution is widely used to identify important model inputs, but evaluating whether these attributions identify truly informative feature

arXiv:2610.04558v1 Announce Type: cross Abstract: We present ClimateBench v2, a standardized protocol for evaluating climate models on diagnostics expected to be informative for their skill in project

Introducing GeoGuess Bench! A benchmark to measure how good AI models are at playing GeoGuessr. We gave each model 210 photos from around the world, asked it to predict where each photo was taken, and

Introducing Hermes Index: an average of benchmarks measuring model performance in Hermes Agent. Our goal with Hermes Index is to better inform users choosing between models and to show labs how they p

arXiv:2610.06469v1 Announce Type: new Abstract: Closed-loop evaluation of end-to-end driving requires continuous rollouts that reveal how earlier decisions affect subsequent driving. However, existing

arXiv:2610.06121v1 Announce Type: new Abstract: Adverse weather and poor illumination remain major challenges for robust ego-trajectory planning in mobile autonomy. 4D radar offers reliable sensing un