Search: “benchmark”

Some strong embeddings models have been released recently. Ours is still better. We benchmarked Embed 5 Fast against Google's EmbeddingGemma…

Some strong embeddings models have been released recently. Ours is still better. We benchmarked Embed 5 Fast against Google's EmbeddingGemma 2 across 163 datasets using RCP-nDCG@10. Embed 5 Fast leads

model-releasescohere--x

17 Oct 2026

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Digital Typhoon Dataset V3: Benchmarking for Multi-spectral Extension and Analyzing Systematic Bias in Intensity Regression

DGX agent

arXiv:2610.21654v1 Announce Type: new Abstract: This paper presents the Digital Typhoon Dataset V3, a new version of the longest typhoon satellite image dataset spanning nearly 50 years, designed to b

model-releasesarxiv-cs-cv

16 Oct 2026

JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs

arXiv:2610.21412v1 Announce Type: new Abstract: Large language model (LLM) agents now fix software, operate computers and serve customers, but their harnesses, the programs that run them, call a large

16 Oct 2026

Turns out le Chonk is into rocket science. Mistral Large 4 scoring well in engineering and astronomy on Ulam’s private benchmark, STEMBench:…

model-releasesmistral-ai--x

Mistral AI announced in mid‑October 2026 that its new model, Mistral Large 4, achieved high scores on Ulam’s private STEM benchmark (STEMBench), particularly excelling in engineering and astronomy cat

16 Oct 2026

When Does 'Post-Call' Begin? An Availability Contract for Earnings-Call NLP Benchmarks

model-releasesarxiv-cs-cl

arXiv:2610.21659v1 Announce Type: cross Abstract: Corporate financial disclosures often unfold over time, with announcements, prepared remarks, and discussion spanning multiple high-frequency return i

16 Oct 2026

Local Ai

DSABench: Benchmarking Source Discovery, Coordination, and Execution for Multi-Database Agents

arXiv:2610.14700v1 Announce Type: new Abstract: Most text-to-SQL benchmarks provide the target database. In a large data catalog, however, choosing the sources is part of the problem: an agent must fi

local-aiarxiv-cs-ai

Frame Evidence:A Benchmark for Reference Frames and Visual Evidence in Multi-Image Spatial Reasoning

arXiv:2610.19564v1 Announce Type: new Abstract: A spatial answer depends on the reference frame specified in language and its supporting visual evidence. Item accuracy alone obscures whether correctne

Hermes Agent is the best place to use the new @NVIDIAAI Nemotron 3.5 Super VL and Bolt models. NVIDIA benchmarked Super VL on Hermes Bench w…

model-releasesnous-research--x

Hermes Agent is the best place to use the new @NVIDIAAI Nemotron 3.5 Super VL and Bolt models. NVIDIA benchmarked Super VL on Hermes Bench with a score of 63.44, and compared to similar local models i

I tested Qwen3.8 27B KLD for 21 Models at 9 different KV quants that fit on 16GB VRAM (189 benchmarks).

model-releasesr-localllama

I calculated KLD and perplexity for Qwen3.8 27B models that fit 16GB VRAM: See comments for the benchmarks. Reddit filters keeps deleting the post. Conclusion: q5_1/q5_1 is almost always better than q

Into the Unknown: Benchmarking AI agents on open scientific research problems

model-releasesarxiv-cs-cl

arXiv:2610.19159v1 Announce Type: cross Abstract: Can AI agents carry out research on open scientific problems? We introduce Into the Unknown, a benchmark that provides an agent with a science paper a

Nvidia's RTX Spark Superchip in the Surface Laptop Ultra outperforms Apple's M5 Max MacBook Pro by ~25% in agentic AI completion benchmarks using Qwen models (Jeffrey Kampman/Tom's Hardware)

Jeffrey Kampman / Tom's Hardware: Nvidia's RTX Spark Superchip in the Surface Laptop Ultra outperforms Apple's M5 Max MacBook Pro by ~25% in agentic AI completion benchmarks using Qwen models — Laying

model-releasestechmeme

QuantEval: A Benchmark for Financial Quantitative Tasks in Large Language Models

model-releasesarxiv-cs-cl

arXiv:2601.08689v3 Announce Type: replace Abstract: Large Language Models (LLMs) have shown strong capabilities in finance-oriented question answering, yet their evaluation on professional quantitativ

UA-Bench: Benchmarking Large Audio-Language Models on Underwater Acoustic Recognition

arXiv:2610.16942v1 Announce Type: cross Abstract: Large audio-language models (LALMs) are fast becoming a general interface to sound, yet how far their broad pretraining carries into specialized domai

UDHRBench: Benchmarking Human Rights in Large Language Models

arXiv:2610.15179v1 Announce Type: cross Abstract: Large language models are increasingly deployed in contexts requiring adherence to ethical and legal norms, yet their alignment with internationally r

CineRecon: Benchmarking Frontier Models for Agentic Cinematic Reconstruction

arXiv:2610.16577v1 Announce Type: new Abstract: Reconstructing reference videos as editable dynamic 3D scenes is an important path toward controllable video generation for filmmaking: these scenes pro

Research

CodeWalk: Generating Coding Benchmarks by Walking a Problem Graph

arXiv:2610.15345v1 Announce Type: new Abstract: High-quality coding problems are expensive to collect. Benchmarks such as SWE-bench rely on substantial human curation, while synthetic alternatives typ

researcharxiv-cs-lg

GIFT-Eval-CTX: A Benchmark For Contextual Forecasting Model Evaluation

arXiv:2610.17794v1 Announce Type: new Abstract: Contextual forecasting seeks to improve time series forecasting by incorporating forward-looking information, such as planned events, external condition

LIBERO-RIG: A Hardware-Setup Robustness Benchmark for Vision-Language-Action Models

arXiv:2610.18843v1 Announce Type: new Abstract: Robot policies are expected to generalize across deployment environments, including variations in the robot itself: cameras are bumped and remounted, ca

VRIQ: Benchmarking and Diagnosing the Visual-Reasoning IQ of Vision-Language Models

arXiv:2602.05382v2 Announce Type: replace Abstract: Vision-language model (VLM) errors on visual reasoning may come from missing visual structure, from mistakes after that structure is extracted, or f

GeoFitBench-LLM: A Benchmark for Assessing Large Language Models' Capabilities in Geospatial Data Requirements Elicitation for Geo-Analytic Tasks

arXiv:2610.12669v1 Announce Type: cross Abstract: Large language models are becoming central to GeoAI systems, yet their evaluation rarely considers the elicitation of geospatial data requirements. Th

12 Oct 2026

🇫🇷 Le Chonk joins the sudo L7 leaderboard 🇫🇷 We just added @MistralAI's Mistral Large 4 to sudo L7, our benchmark for staff-level softwa…

model-releasesarthur-mensch--x

🇫🇷 Le Chonk joins the sudo L7 leaderboard 🇫🇷 We just added @MistralAI's Mistral Large 4 to sudo L7, our benchmark for staff-level software engineering. Mistral Large 4 scores 13.3%, ahead of Muse Spar

12 Oct 2026

What Should Embodied Agents Remember? Benchmarking and Evaluating Memory Modality for Embodied Agents

arXiv:2610.12561v1 Announce Type: new Abstract: Memory allows large language model (LLM) agents to use past experience, often stored as textual records, when responding to requests. For embodied agent

12 Oct 2026

CityDeploy-Bench: Benchmarking Physics-Grounded Spatial Set Planning for Multi-Transmitter Network Deployment

arXiv:2610.11065v1 Announce Type: new Abstract: Automating city-scale wireless deployment remains challenging under complex urban propagation and network-wide interference. We introduce extbf{CityDepl

Elucidating the Space of Enzymatic Reaction: A Unified Benchmark and Pretrained Model

arXiv:2610.11694v1 Announce Type: cross Abstract: Existing reaction models primarily learn molecular transformations, whereas enzy- matic reactions depend jointly on molecular structure and catalytic

Humanity's Sixth Sense: Benchmarking Intuitive Visual Reasoning in Multimodal Models

arXiv:2610.08966v2 Announce Type: replace Abstract: Humans perceive far more in a scene than what is explicitly depicted: a single glance captures past causes and future trajectories; a quick peek det

ManiUnit: A Manipulation Skill Dataset and Benchmark for Long-Horizon Tasks

arXiv:2610.12089v1 Announce Type: new Abstract: Long-horizon mobile manipulation requires a robot to navigate multi-room environments and execute a sequence of manipulation skills under a single natur

MissMAC-Bench: Building Solid Benchmark for Missing Modality Issue in Robust Multimodal Affective Computing

arXiv:2602.00811v2 Announce Type: replace Abstract: Current Multimodal Affective Computing (MAC) systems heavily rely on the completeness of multiple modalities to accurately understand human's affect

Searching for 'Harmful Refusal': A Psychometric Audit of an AI Safety Benchmark

arXiv:2610.12409v1 Announce Type: new Abstract: Safety benchmarks typically report one overall score for a suite of datasets, each of which may target one or more safety-related attributes, so models

Stepping away from Benchmarks and Code, what models are you using for Creating writing projects

model-releasesr-localllama

All anyone ever talks in this sub is about x new model having x benchmark scores and how well it can code. Or some new tech to increase t/s for qwen. So I wanted to go back to the roots of this sub an

AdSpark: A Large-Scale Dataset and Benchmark for Product-Centric Advertisement Video Generation

arXiv:2610.10047v1 Announce Type: new Abstract: Product-centric advertisement video generation aims to create promotional videos that preserve fine-grained product identity while presenting selling po

Best practices when running a benchmark on online models [D]

model-releasesr-machinelearning

I'm developing a benchmark for a low resource language and I don't want it to be leaked and used for training when it is being used to get predictions. For locally run models it shouldn't be a problem

Research

HomID : Benchmarking Intrinsic Dimension Estimators on Homogenous Manifolds with Anisotropic Embeddings

arXiv:2510.01335v2 Announce Type: replace Abstract: The manifold hypothesis suggests that data lies on manifolds with smaller intrinsic dimension (ID) than their ambient dimension. However there is no

researcharxiv-cs-lg

Lifelong small-object navigation in changing object layouts: a benchmark and method

arXiv:2610.10125v1 Announce Type: cross Abstract: Household robots need to continually navigate to different objects in the same environment, many of which are small and portable, such as tools and to

An Information-Theoretic Evaluation Framework for Benchmark and Model Diagnosis in Knowledge Tracing

arXiv:2610.06988v1 Announce Type: new Abstract: Knowledge tracing (KT) models are predominantly evaluated using aggregate metrics such as area under the curve (AUC) and accuracy. However, these global

Benchmarking Time Series Foundation Models for Load Forecasting Under Covariate Uncertainty

arXiv:2610.07232v1 Announce Type: new Abstract: Accurate short-term load forecasting (STLF) is essential for the reliable and efficient operation of modern power systems. While time series foundation

Nine Trials to Recover: A Reproducible Benchmark for Repertoire-Free Soft-Robot Damage Adaptation

arXiv:2610.07616v1 Announce Type: new Abstract: We present a repertoire-free benchmark for soft-robot damage recovery under nine online trials. The protocol pairs damage masks within each morphology,

Offline AI Modules: Voice-First Offline Architecture, Hardware Reference Stack, Quantization and Benchmarking

arXiv:2610.07026v1 Announce Type: new Abstract: The Offline AI Modules workstream enables practical, low-power, and community-accessible deployment of voice-first AI systems that operate fully offline

RenderBench: Benchmarking Render-to-Real Video Transfer with Reconstructed Digital Twins

arXiv:2610.08684v1 Announce Type: new Abstract: Modern video models can generate realistic videos from real appearance references and proxy renders that specify scene structure, viewpoint changes, and

Symphony for Text Generation: Benchmarking Clinical Note Generation

arXiv:2610.08161v1 Announce Type: cross Abstract: Ambient documentation systems are rapidly gaining adoption, yet their impact on clinical note quality remains poorly characterized. We introduce MedCo

Towards an Extensible Benchmark for Spoken Dialogue with Social Robots

arXiv:2610.08733v1 Announce Type: new Abstract: Language models provide a plug-and-play interface between humans and robots, but important challenges remain when speech, dialogue, fast interaction, an

Towards benchmarking Western Bluebird detection in the wild

arXiv:2610.07802v1 Announce Type: new Abstract: Bird monitoring in natural environments is challenging due to the small size of some species of birds relative to the scene, background clutter, variabi

CHAOSMINING: Benchmarking Post-Hoc Attribution with Sparse Informative Features in High Dimensions

arXiv:2406.12150v2 Announce Type: replace Abstract: Post-hoc attribution is widely used to identify important model inputs, but evaluating whether these attributions identify truly informative feature

ClimateBench v2.0: Probabilistic Climate Model Benchmarking

arXiv:2610.04558v1 Announce Type: cross Abstract: We present ClimateBench v2, a standardized protocol for evaluating climate models on diagnostics expected to be informative for their skill in project

Introducing GeoGuess Bench! A benchmark to measure how good AI models are at playing GeoGuessr. We gave each model 210 photos from around th…

model-releasestogether-ai--x

Introducing GeoGuess Bench! A benchmark to measure how good AI models are at playing GeoGuessr. We gave each model 210 photos from around the world, asked it to predict where each photo was taken, and

Agents

Introducing Hermes Index: an average of benchmarks measuring model performance in Hermes Agent. Our goal with Hermes Index is to better info…

Introducing Hermes Index: an average of benchmarks measuring model performance in Hermes Agent. Our goal with Hermes Index is to better inform users choosing between models and to show labs how they p

agentsnous-research--x

Odyssey: A Closed-Loop Benchmark for Long-Horizon Real-World Driving with Explicit Navigation Routes

arXiv:2610.06469v1 Announce Type: new Abstract: Closed-loop evaluation of end-to-end driving requires continuous rollouts that reveal how earlier decisions affect subsequent driving. However, existing

Radar2Plan: Benchmarking 4D Radar for End-to-End Open-Loop Ego-Trajectory Planning

arXiv:2610.06121v1 Announce Type: new Abstract: Adverse weather and poor illumination remain major challenges for robust ego-trajectory planning in mobile autonomy. 4D radar offers reliable sensing un

SWE-Race: a coding-agent benchmark of 188 real concurrency bugs, with results from three models [P]

model-releasesr-machinelearning

We've been building a benchmark out of real concurrency bugs (race conditions, deadlocks, cancellation issues) taken from merged PRs in about 100 Python projects. Each task gets graded by the project'