Search: “benchmark”

WorldReasonBench: Human-Aligned Stress Testing of Video Generators as Future World-State Predictors

arXiv:2605.10434v1 Announce Type: new Abstract: Commercial video generation systems such as Seedance2.0 and Veo3.1 have rapidly improved, strengthening the view that video generators may be evolving i

12 May 2026

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?

DGX agent

arXiv:2605.07937v1 Announce Type: new Abstract: Long-horizon AI agents execute complex workflows spanning hundreds of sequential actions, yet a single wrong assumption early on can cascade into irreve

model-releasesarxiv-cs-cl

11 May 2026

CarCrashNet: A Large-Scale Dataset and Hierarchical Neural Solver for Data-Driven Structural Crash Simulation

arXiv:2605.07098v1 Announce Type: new Abstract: Crash simulation is a cornerstone of modern vehicle development because it reduces the need for costly physical prototypes, accelerates safety-driven de

Dataset Watermarking for Closed LLMs with Provable Detection

arXiv:2605.06865v1 Announce Type: new Abstract: Large language models (LLMs) are pre-trained and post-trained on vast amounts of loosely curated data, raising the possibility that these models may hav

HyperEyes: Dual-Grained Efficiency-Aware Reinforcement Learning for Parallel Multimodal Search Agents

arXiv:2605.07177v1 Announce Type: cross Abstract: Existing multimodal search agents process target entities sequentially, issuing one tool call per entity and accumulating redundant interaction rounds

ModelLens: Finding the Best for Your Task from Myriads of Models

arXiv:2605.07075v1 Announce Type: new Abstract: The open-source model ecosystem now contains hundreds of thousands of pretrained models, yet picking the best model for a new dataset is increasingly in

PolySQL: Scaling Text-to-SQL Evaluation Across SQL Dialects via Automated Backend Isomorphism

arXiv:2605.07796v1 Announce Type: new Abstract: SQL dialects vary in syntax, types, and functions across database engines. Text-to-SQL benchmarks, however, predominantly support only SQLite. This crea

RuleSafe-VL: Evaluating Rule-Conditioned Decision Reasoning in Vision-Language Content Moderation

arXiv:2605.07760v1 Announce Type: new Abstract: Platform content moderation applies explicit policy rules and context-dependent conditions to decide whether user content is allowed, restricted, or rem

S2S-Arena: Evaluating Paralinguistic Instruction Following in Speech-to-Speech Models

arXiv:2503.05085v2 Announce Type: replace Abstract: Recent advances in large language models (LLMs) have fundamentally reshaped speech-to-speech (S2S) systems, enabling increasingly natural spoken int

Tracing Uncertainty in Language Model 'Reasoning'

arXiv:2605.07776v1 Announce Type: cross Abstract: Language model (LM) 'reasoning', commonly described as Chain-of-Thought or test-time scaling, often improves benchmark performance, but the dynamics u

Detailed review and guide from my testing of local ollama setup with DeepSeek models (Ryzen APU's only)

This post provides a detailed review and practical guide for setting up and testing Ollama with DeepSeek models specifically on Ryzen APU systems. It likely covers performance benchmarks, configuratio

model-releasesr-ollama

10 May 2026

Jarred tried rewriting Bun in Rust and it passes 99.8% of the existing test suite we're not being ambitious enough

Jarred tried rewriting Bun in Rust and it passes 99.8% of the existing test suite we're not being ambitious enough there will be a blog post about this. on what this means for bun, benchmarks, memory

model-releasesthariq--x

10 May 2026

Agents

Using MemAlign to Improve Evaluation of Traditional Machine Learning in Genie Code

MemAlign is a technique used within Databricks' Genie code environment to enhance the evaluation and benchmarking of traditional machine learning models by optimizing memory alignment. This approach l

agentsdatabricks

8 May 2026

A Comparative Study of PyCaret AutoML and CNN-BiLSTM for Binary Hate Speech Detection in Indonesian Twitter

arXiv:2605.04885v1 Announce Type: new Abstract: This paper compares a PyCaret AutoML branch and a CNN-BiLSTM branch for binary hate speech detection on Indonesian Twitter using the HS label from the c

Agent harnesses have an expiration date

A benchmark-driven look at why agent harnesses need adaptive finish logic as model behavior changes across Claude, GPT-4o, and Gemma. The post Agent harnesses have an expiration date appeared first on

model-releasesarize-ai

AsymmetryZero: A Framework for Operationalizing Human Expert Preferences as Semantic Evals

arXiv:2605.04083v1 Announce Type: new Abstract: Much of the focus in RL today is on evaluation design: building meaningful evals that serve simultaneously as benchmarks and as well-defined reward sign

Capacity-Aware Mixture Law Enables Efficient LLM Data Optimization

arXiv:2603.08022v2 Announce Type: replace Abstract: A data mixture refers to how different data sources are combined to train large language models, and selecting an effective mixture is crucial for o

Implicit Representations of Grammaticality in Language Models

arXiv:2605.05197v1 Announce Type: new Abstract: Grammaticality and likelihood are distinct notions in human language. Pretrained language models (LMs), which are probabilistic models of language fitte

Safety

Misaligned by Reward: Socially Undesirable Preferences in LLMs

arXiv:2605.05003v1 Announce Type: new Abstract: Reward models are a key component of large language model alignment, serving as proxies for human preferences during training. However, existing evaluat

safetyarxiv-cs-cl

Privacy-Preserving Empathy Detection in Video Interactions

arXiv:2504.10808v3 Announce Type: replace Abstract: Detecting empathy from video interactions has emerging applications, yet raw videos that could be used for training AI models are rarely available d

Agents

Revisiting the Travel Planning Capabilities of Large Language Models

arXiv:2605.03308v1 Announce Type: new Abstract: Travel planning serves as a critical task for long-horizon reasoning, exposing significant deficits in LLMs. However, existing benchmarks and evaluation

agentsarxiv-cs-ai

ROCm Status in mid 2026 [D]

researchr-machinelearning

ROCm has reached production-ready status for PyTorch and vLLM workloads in 2026. PyTorch runs well and MI300X benchmarks are competitive, though the ecosystem gap with CUDA remains real. AMD's MI355X

ABD: Default Exception Abduction in Finite First Order Worlds

arXiv:2602.18843v3 Announce Type: replace Abstract: We introduce ABD, a benchmark for default-exception abduction over finite first-order worlds. Given a background theory with an abnormality predicat

Effective Performance Measurement: Challenges and Opportunities in KPI Extraction from Earnings Calls

arXiv:2605.03147v1 Announce Type: new Abstract: Earnings calls are a key source of financial information about public companies. However, extracting information from these calls is difficult. Unlike t

RoboEval: Where Robotic Manipulation Meets Structured and Scalable Evaluation

arXiv:2507.00435v2 Announce Type: replace-cross Abstract: We introduce RoboEval, a structured evaluation framework and benchmark for robotic manipulation that augments binary success with principled b

Sentiment Analysis of Indonesian Spotify Reviews Using Machine Learning and BiLSTM

arXiv:2605.03443v1 Announce Type: new Abstract: This paper benchmarks classical machine learning and deep learning approaches for three-class sentiment classification of Indonesian Spotify reviews. Us

Agents

The Conversations Beneath the Code: Triadic Data for Long-Horizon Software Engineering Agents

arXiv:2605.02244v1 Announce Type: cross Abstract: Frontier software engineering agents have saturated short-horizon benchmarks while regressing on the work that constitutes senior engineering: long-ho

agentsarxiv-cs-ai

Valley3: Scaling Omni Foundation Models for E-commerce

arXiv:2605.01278v1 Announce Type: new Abstract: In this work, we present Valley3, an omni multimodal large language model (MLLM) developed for diverse global e-commerce tasks, with unified understandi

Chart-FR1: Visual Focus-Driven Fine-Grained Reasoning on Dense Charts

arXiv:2605.01882v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) have shown considerable potential in chart understanding and reasoning tasks. However, they still struggle with

Coopetition-Gym v1: A Formally Grounded Platform for Mixed-Motive Multi-Agent Reinforcement Learning under Strategic Coopetition

arXiv:2605.02063v1 Announce Type: cross Abstract: We present Coopetition-Gym v1, a benchmark platform for mixed-motive multi-agent reinforcement learning under strategic coopetition. The platform comp

Creating and Evaluating Figurative Language Dataset for Sindhi

arXiv:2605.01323v1 Announce Type: new Abstract: In this article, we introduce SiNFluD, a novel benchmark dataset for Sindhi figurative language classification. We first collect raw text from various b

EditPropBench: Measuring Factual Edit Propagation in Scientific Manuscripts

arXiv:2605.02083v1 Announce Type: new Abstract: Local factual edits in scientific manuscripts often create non-local revision obligations. If a dataset changes from 215 to 80 documents, claims such as

Enhanced LLM Reasoning by Optimizing Reward Functions with Search-Driven Reinforcement Learning

arXiv:2605.02073v1 Announce Type: new Abstract: Mathematical reasoning is a key benchmark for large language models. Reinforcement learning is a standard post-training mechanism for improving the reas

GEODE: Angle-Adaptive OOD Detection with Universal Scorer Compatibility

arXiv:2605.01063v1 Announce Type: cross Abstract: Outlier Exposure (OE) is among the strongest training-based OOD detectors on standard benchmarks but exhibits scorer-dependent tradeoffs (e.g., strong

researcharxiv-cs-cv

LITcoder: A General-Purpose Library for Building and Comparing Encoding Models

arXiv:2509.09152v2 Announce Type: replace Abstract: We introduce LITcoder, an open-source library for building and benchmarking neural encoding models. Designed as a flexible backend, LITcoder provide

Multi-Perspective Transformers in ARC-AGI-2 Challenge

arXiv:2605.01154v1 Announce Type: new Abstract: ARC-AGI-2 is a benchmark of human-intuitive visual puzzles that measures a machine's ability to generalize from limited examples, interpret symbolic mea

SurgCheck: Do Vision-Language Models Really Look at Images in Surgical VQA?

arXiv:2605.01911v1 Announce Type: new Abstract: Purpose: Vision-language models (VLMs) have shown promising performance in surgical visual question answering (VQA). However, existing surgical VQA data

Towards Visual Query Localization in the 3D World

arXiv:2605.01498v1 Announce Type: new Abstract: Visual query localization (VQL) aims to predict the spatio-temporal response of the most recent occurrence in a sequence given a query. Currently, most

AgentFloor: How Far Up the tool use Ladder Can Small Open-Weight Models Go?

arXiv:2605.00334v1 Announce Type: cross Abstract: Production agentic systems make many model calls per user request, and most of those calls are short, structured, and routine. This raises a practical

Can Coding Agents Reproduce Findings in Computational Materials Science?

arXiv:2605.00803v1 Announce Type: cross Abstract: Large language models are increasingly deployed as autonomous coding agents and have achieved remarkably strong performance on software engineering be

RouteProfile: Elucidating the Design Space of LLM Profiles for Routing

arXiv:2605.00180v1 Announce Type: cross Abstract: As the large language model (LLM) ecosystem expands, individual models exhibit varying capabilities across queries, benchmarks, and domains, motivatin

Applications

That doesn't mean that there should not be regulation and vetting, but it does suggest that is hard to write a criteria right now that is no…

applicationsethan-mollick--x

That doesn't mean that there should not be regulation and vetting, but it does suggest that is hard to write a criteria right now that is not somewhat vague. More R&D into non-lab benchmarks is urgent

Safety

Agent-Agnostic Evaluation of SQL Accuracy in Production Text-to-SQL Systems

arXiv:2604.28049v1 Announce Type: new Abstract: Text-to-SQL (T2SQL) evaluation in production environments poses fundamental challenges that existing benchmarks do not address. Current evaluation metho

safetyarxiv-cs-ai

BatteryPass-12K: The First Dataset for the Novel Digital Battery Passport Conformance Task

arXiv:2604.26986v1 Announce Type: new Abstract: We introduce a novel task of digital battery passport (DBP) conformance classification and introduce the first public benchmark for the task: BatteryPas

Beyond the Training Distribution: Mapping Generalization Boundaries in Neural Program Synthesis

arXiv:2604.27551v1 Announce Type: cross Abstract: Large-scale transformers achieve impressive results on program synthesis benchmarks, yet their true generalization capabilities remain obscured by dat

researcharxiv-cs-ai

CausalCompass: Evaluating the Robustness of Time-Series Causal Discovery in Misspecified Scenarios

arXiv:2602.07915v2 Announce Type: replace-cross Abstract: Causal discovery from time series is a fundamental task in machine learning. However, its widespread adoption is hindered by a reliance on unt

From Unstructured Recall to Schema-Grounded Memory: Reliable AI Memory via Iterative, Schema-Aware Extraction

arXiv:2604.27906v1 Announce Type: new Abstract: Persistent AI memory is often reduced to a retrieval problem: store prior interactions as text, embed them, and ask the model to recover relevant contex

Make sure to read the blog post for a detailed analysis of frontier model failure modes: https://arcprize.org/blog/arc-agi-3-gpt-5-5-opus-4-…

model-releasesfrancois-chollet--x

Francois Chollet shared a blog post analyzing failure modes of frontier AI models, specifically examining performance on the ARC (Abstraction and Reasoning Corpus) AGI benchmark with models including