Search: “benchmark”

BrickBench: Evaluating Agentic Brick Design

arXiv:2610.12452v1 Announce Type: new Abstract: We propose BrickBench, a benchmark for agentic text-conditioned LEGO-set design. Given a prompt, an agent is tasked with producing an assembly that not

Obscuring Data Contamination Through Translation: Evidence from Arabic Corpora

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

DGX agent

arXiv:2601.14994v2 Announce Type: replace-cross Abstract: Data contamination can invalidate benchmark evaluation when a model benefits from memorized evaluation content rather than genuine generalizat

model-releasesarxiv-cs-ai

9 Oct 2026

Reasoning-Informed Visual Editing

arXiv:2610.12343v1 Announce Type: new Abstract: Large Multi-modality Models (LMMs) have made significant progress in visual understanding and generation, but still face challenges in visual editing, p

SWE-Journey: Towards More Realistic Evaluation of Coding Assistants through Long-Horizon, Multi-Turn Interaction

arXiv:2610.11559v1 Announce Type: cross Abstract: Coding assistants such as Claude Code and Codex have become a major application of LLM agents, yet existing benchmarks remain far from real-world use,

WOVEN: Weaving Visual World Modeling into Multimodal LLMs

arXiv:2610.12417v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) struggle with spatial, embodied, physical, and temporal reasoning. We hypothesize that these failures reflect

Conditional Accuracy Profiles: Diagnosing LLM Judges across Deployment Conditions

arXiv:2610.09229v1 Announce Type: new Abstract: LLM-as-judge is now a standard tool for scalable evaluation, but judge performance is still often summarized by a single accuracy number. This aggregate

LLM4Impact: Integrating Heterogeneous Information for Scientific Impact Prediction

arXiv:2610.10138v1 Announce Type: new Abstract: Predicting the future impact of a newly published paper is challenging because it must be inferred from heterogeneous evidence available at publication

Quad-State Safety Evaluation of Open-Weight Large Language Models on Non-Canonical Inputs

arXiv:2610.09033v1 Announce Type: new Abstract: Standard safety evaluations of large language models assess harmful requests written in canonical plain text, while models in real-world deployment rout

VideoZeroBench: Probing the Limits of Video MLLMs with Spatio-Temporal Evidence Verification

arXiv:2604.01569v2 Announce Type: replace Abstract: Video multimodal large language models achieve strong results on existing benchmarks, but answer accuracy alone does not establish whether they can

BoT-Feedback: Grounding Multimodal Reasoning in Biomechanical Evidence for Explainable Human Action Feedback

arXiv:2610.06972v1 Announce Type: new Abstract: Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities in visual understanding and multimodal reasoning, yet they remain fun

CroissantMiner: Automated Extraction and Validation of Croissant Metadata for ML Datasets

arXiv:2610.07132v1 Announce Type: cross Abstract: Croissant has emerged as a standard for machine-readable dataset metadata, yet populating its fields remains labor-intensive and requires careful read

Harness Engineering for Software Engineering via Modular Executable Dev-Primitives

arXiv:2610.07832v1 Announce Type: cross Abstract: Large language models (LLMs) equipped with terminal access have demonstrated strong capabilities in automating software engineering tasks. However, ex

Image Bitstream Fine-grained Understanding for Privacy-Friendly AIoT

arXiv:2610.08414v1 Announce Type: new Abstract: Image Bitstream Fine-grained Understanding (IBFU) aims to directly perform fine-grained classification and semantic description generation from encoded

Measuring climate backlash in Twitter and Reddit archives: Lexical definitions, recorded responses and participant turnover

arXiv:2610.07634v1 Announce Type: new Abstract: Social media archives are often used to study resistance to climate action, but words, response counters and observed participants do not measure the sa

Medical Image Alignment Assessment as a Test of Generalist Visual Reasoning in Frontier Multimodal Models

arXiv:2610.06896v1 Announce Type: new Abstract: Frontier multimodal large language models (MLLMs) are increasingly positioned as general purpose visual reasoners as part of the quest for artificial ge

MemMux: Runtime Verification and Honest Resource Attribution for Fleets of Parallel Coding Agents

arXiv:2610.07257v1 Announce Type: new Abstract: Developers increasingly run a fleet of coding agents side by side on one workstation. The tools they reach for, terminal multiplexers like tmux and a ne

Same-Number Citation Swaps: Stress-Testing Jev as a Financial Evidence Judge

arXiv:2610.08675v1 Announce Type: new Abstract: Financial reports repeat values across periods, metrics and accounting lines, allowing an LLM-generated calculation to be numerically correct while citi

Seeing Through the Displaced Frame: Privileged Noise Distillation for Vision-Force Precision Assembly

model-releasesarxiv-cs-ro

arXiv:2610.07745v1 Announce Type: new Abstract: Pose error in precision assembly can corrupt not only what a robot observes but also the coordinate frame in which it acts. On the FORGE benchmark, the

Too Categorical to be Human: Emotion Concepts in LLMs and Humans

arXiv:2508.05880v3 Announce Type: replace-cross Abstract: Understanding human emotions is central to user-facing AI applications, safety alignment, and the simulation of human behavior. As emotional s

WorldSolver: Can LLM Agents Simulate the Physical Dynamics via Solver Generation?

arXiv:2610.08720v1 Announce Type: new Abstract: LLM-based agents are increasingly advancing scientific and engineering problem solving, with physics simulation emerging as a challenging yet practical

zkLLMPoT: Efficient Zero Knowledge Proof of Training for Large Language Models

arXiv:2610.08258v1 Announce Type: new Abstract: Auditing the claimed outcomes of large language model (LLM) training is challenging when model weights and training data are private, while cryptographi

BrainTRACE: Tracing Longitudinal, Multimodal, and Volumetric Evidence in Brain MRI Clinical Reasoning

arXiv:2610.06571v1 Announce Type: new Abstract: Brain MRI interpretation is a longitudinal clinical reasoning problem: radiologists compare serial studies, integrate information across MRI sequences,

DEFAME: Dynamic Evidence-based FAct-checking with Multimodal Experts

arXiv:2412.10510v5 Announce Type: replace Abstract: The proliferation of disinformation demands reliable and scalable fact-checking solutions. We present Dynamic Evidence-based FAct-checking with Mult

RAC: Rectified Flow Auto Coder

arXiv:2603.05925v2 Announce Type: replace Abstract: In this paper, we propose a Rectified Flow Auto Coder (RAC) inspired by Rectified Flow to replace the traditional VAE: 1. It achieves multi-step dec

What Do Verifiable Rewards Teach Video-Language Models About Time? A Controlled Multi-Model Study

arXiv:2610.03792v1 Announce Type: new Abstract: Reinforcement learning from verifiable rewards (RLVR) has produced large reasoning gains in language models, and verifiable video benchmarks make it app

An automated pipeline for standardised speech-unit annotation in spontaneous dialogue

arXiv:2610.03078v1 Announce Type: new Abstract: Quantifying conversational dynamics requires reliable identification of interactional units and their temporal boundaries, but speech activity alone doe

Automating the Application of HCI Principles: Skills for On-Demand UI Construction, the Human-AI Space to Think, and the Future of HCI

arXiv:2610.02369v1 Announce Type: cross Abstract: Human-computer interaction (HCI) is in the middle of a transition: large language models can now generate functional user interfaces (UIs) on demand f

Can AI Understand the Language of Origami?

arXiv:2603.13856v3 Announce Type: replace-cross Abstract: Building AI systems that can plan, act, and create in the physical world requires more than pattern recognition. Such systems must reason abou

Safety

DEPICT: Scoring Text-to-Image Alignment by Answer Agreement

arXiv:2610.03617v1 Announce Type: new Abstract: Image-text alignment is a core problem in computer vision with applications in caption evaluation, hallucination detection, data curation, and the bench

safetyarxiv-cs-cv

DISSOLVR: An Interpretable and Fast Framework for Aqueous and Organic Solubility Prediction

arXiv:2610.02574v1 Announce Type: new Abstract: High-fidelity solubility prediction is fundamental to pharmaceutical development and environmental partitioning, where accurate modeling must couple mol

From Retrieval to Typed Decisions: Calibrated System One Models from Biomedical Sentence Encoders

arXiv:2610.02486v1 Announce Type: cross Abstract: Typed decision models answer schema-constrained questions about a text in one forward pass and return probabilities meant to be thresholded. We ask wh

Latent-MOPD: Latent Multi-Teacher On-Policy Distillation

arXiv:2610.02381v1 Announce Type: new Abstract: On-policy distillation (OPD) trains a student on the responses it generates. Existing LLM multi-teacher OPD transfers what specialists predict through t

TerraVis: Towards Evaluation of World-Grounded Visual Consistency in Text-to-Image Generation via MLLM Workflows

arXiv:2610.02959v1 Announce Type: new Abstract: Recent text-to-image models have made substantial progress in photorealism, aesthetics, and text-image alignment. Yet visually appealing images can stil

When Predicting Nothing Beats SAM 3: Revisiting Evaluation in Video Object Segmentation

arXiv:2610.02946v1 Announce Type: new Abstract: Video Object Segmentation (VOS) in complex and long videos is increasingly important for real-world applications, where target objects often appear only

Agents Are Systems, Not Models: Rethinking Agentic Evaluation

arXiv:2610.01618v1 Announce Type: new Abstract: Agent evaluations increasingly go beyond a single success rate, reporting metrics such as cost, consistency, and robustness. Yet they typically treat th

Capabilities Ain't All You Need: Measuring Propensities in AI

arXiv:2602.18182v5 Announce Type: replace-cross Abstract: AI evaluation has primarily focused on measuring capabilities, with formal approaches inspired from Item Response Theory (IRT) being increasin

DiVid: Diagnosing Dimension-Specific Diversity Collapse in Video Generation Models

arXiv:2610.01661v1 Announce Type: new Abstract: Despite remarkable progress, video generation models often produce highly similar outputs when repeatedly sampled from the same prompt, limiting their u

Do MLLM Judges Judge the Edit? Auditing Bias in Image Editing Evaluation with Verified Quality Preservation

arXiv:2610.01670v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) are increasingly used as automated judges for instruction-based image editing and as reward signals for model t

Ego2Act: Evaluating Goal-Directed Manipulation in Egocentric Video Generation

arXiv:2610.01092v1 Announce Type: new Abstract: Video generation models are increasingly being explored as world simulators for embodied planning and learning. To do so effectively, these models must

Agents

From Reasoning Failures to Composable Video Spatial Intelligence

arXiv:2610.01999v1 Announce Type: new Abstract: Spatial reasoning benchmarks evaluate vision-language models across diverse tasks, but task-level scores do not reveal which underlying capabilities acc

agentsarxiv-cs-cv

Harnessing Vision-Language Models for Perceptual Quality Assessment and Autonomous Content Adjustment in Augmented Reality

arXiv:2610.00677v1 Announce Type: new Abstract: Advancements in augmented reality (AR) continue to foster innovative solutions, facilitating novel methodologies within educational systems, healthcare

Kepler: Auditable World Models for ARC-AGI-3

arXiv:2610.00834v1 Announce Type: new Abstract: ARC-AGI-3 evaluates agents in interactive environments whose rules and objectives must be inferred from observation. We present Kepler, an open-source h

Safety

More with Less: a Large Scale Remote Sensing VLM with a Simple Recipe

arXiv:2607.15942v2 Announce Type: replace Abstract: Remote sensing vision-language models are increasingly expected to support open-ended reasoning over Earth Observation data and a variety of tasks.

safetyarxiv-cs-cv

PACE: Provenance-Aware Capability Enforcement for Tool-Using LLM Agents

arXiv:2610.01349v1 Announce Type: cross Abstract: Tool-using large language model (LLM) agents turn generated text into real side effects, so poisoned tool metadata, retrieved pages, memory, and reusa

Permutation-Robust Decision Modeling with Candidate-Independent Block-Causal Attention

arXiv:2610.01601v1 Announce Type: cross Abstract: Decision models often score a variable-sized set of candidate actions encoded in a single sequence. This setting is increasingly relevant for System 1

Agents

ReLiveGym: Evaluating Long-Lived Agents over Weeks of Replayed Reality

arXiv:2610.00710v1 Announce Type: new Abstract: As large language model (LLM) agents become widely adopted, they are increasingly deployed for tasks that require persistent monitoring or recurring act

agentsarxiv-cs-ai

Same Reward, Different Skills: When Multimodal RL Learns to Look

arXiv:2610.01908v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) improves vision-language benchmark scores even without visual information during training. With im

The Null Is the Hard Part: Exact Tests for Memorization in Generative Models

arXiv:2610.00251v1 Announce Type: new Abstract: Memorization audits of generative models read similarity scores against thresholds, with no null distribution, and the conclusions they support can be w