Search: “benchmark”

CrackedPDFs: A Controlled Benchmark for Hidden Prompt Injection in PDFs

arXiv:2607.19396v1 Announce Type: new Abstract: Document-based LLM systems often flatten a PDF before guardrails inspect it. That step can discard evidence that an instruction was never visible to the

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

CruiseBench: A Real-Flight-Aligned N-CMAPSS Benchmark for Engine RUL Prediction

DGX agent

arXiv:2607.19380v1 Announce Type: new Abstract: Remaining useful life (RUL) prediction estimates how long an engine can continue safe operation and is central to maintenance planning. N-CMAPSS extends

model-releasesarxiv-cs-lg

23 Jul 2026

ExpertVerse: A General-Purpose Benchmark for Expert-Level Reasoning in Knowledge-Intensive Visual Synthesis

arXiv:2607.19341v1 Announce Type: new Abstract: Recent advances in multimodal generative models have enabled instruction-based image generation to move beyond semantic manipulation to knowledge-driven

Learning Explicit Physical Parameter Control and Benchmarking for Video Generation

arXiv:2607.18924v1 Announce Type: new Abstract: Recent advances in image-to-video generation have improved visual realism, making physically grounded and controllable dynamics an important step toward

NavVerse: Benchmarking Indoor-to-Outdoor Embodied Navigation in Continuous Robot Simulation

model-releasesarxiv-cs-ro

arXiv:2607.19695v1 Announce Type: new Abstract: Robots deployed in delivery, campus, and emergency-response settings often need to navigate from buildings to streets within a single continuous episode

Open-Vocabulary Gaze Object Prediction: Benchmark and Method

arXiv:2607.18827v1 Announce Type: new Abstract: Gaze Object Prediction (GOP) aims to localize and recognize the objects humans attend to, a task crucial for understanding human-centric interactions. H

OpenSkillRisk: Benchmarking Agent Safety When Using Real-World Risky Third-Party Skills

arXiv:2607.20121v1 Announce Type: new Abstract: LLM-based agents leverage third-party skills to extend their capabilities in open-world scenarios. However, third-party skills can introduce extra secur

PhenSPINE: A Standardized Benchmark for Spine Pathology Diagnosis

arXiv:2607.19696v1 Announce Type: cross Abstract: The accurate diagnosis of spinal pathologies depends heavily on radiological interpretation, yet automated systems are hindered by the lack of diverse

DevicesWorld: Benchmarking Cross-Device Agents in Heterogeneous Environments

arXiv:2607.13465v1 Announce Type: cross Abstract: LLM-based agents have rapidly improved at operating individual digital environments such as mobile applications, desktop systems, and smart homes. How

16 Jul 2026

nuTruck: Benchmarking Autonomous Driving Planning for Distributed Electric-drive Trucks

model-releasesarxiv-cs-ro

arXiv:2607.13704v1 Announce Type: new Abstract: The dominance of traditional rule-based methods in autonomous driving has gradually been replaced by learning-based approaches. While learning-based pla

16 Jul 2026

RAGthoven at SemEval-2026 Task 1: A Multi-Stage Pipeline Walks Into a Benchmark and Barely Clears the Bar

arXiv:2607.13189v1 Announce Type: cross Abstract: We present RAGthoven, our system for SemEval-2026 Task 1 (MWAHAHA), Subtask A (multilingual constrained humor generation in English, Spanish, and Chin

16 Jul 2026

Safety

Operationalising Multi-Dimensional Evaluation for Conversational Agents: A Scalable, Governed Pipeline with Selective Re-evaluation and Model Benchmarking

arXiv:2607.12085v1 Announce Type: new Abstract: Evaluating retail conversational agents requires methods beyond lexical-overlap metrics to assess intent alignment, factuality, helpfulness, clarity, to

safetyarxiv-cs-ai

15 Jul 2026

HumanForge: A Human-Centric Deepfake Video Benchmark with Multi-Agent Forgery Rationales

arXiv:2607.08705v1 Announce Type: new Abstract: Rapid advancements in video diffusion models and temporal editing tools have enabled the generation of highly realistic human-centric videos, posing unp

10 Jul 2026

UniRef-UAV: A Multimodal Benchmark for Universal Referring in UAV Imagery

arXiv:2607.08267v1 Announce Type: new Abstract: Unmanned aerial vehicles (UAVs) increasingly rely on visual grounding capabilities to localize task-relevant targets from diverse instructions in comple

10 Jul 2026

Does AI Understand Imaging? A Systematic Benchmark of Agentic AI for Computational Imaging Tasks

arXiv:2607.07189v1 Announce Type: new Abstract: Vision-language models (VLMs) and agentic AI have shown strong performance on semantic visual tasks, but it remains unclear whether they can handle the

9 Jul 2026

Generalist Vision-Language Models for Fast Radio Burst detection: a zero-shot benchmark against a specialized detector

model-releasesarxiv-cs-lg

arXiv:2607.07382v1 Announce Type: new Abstract: Fast Radio Bursts (FRBs) are millisecond-duration radio transients whose automated detection increasingly relies on highly specialized deep learning mod

9 Jul 2026

HAJJv2-CrowdCount: Zero-Shot Benchmark for Dense Crowd Counting

arXiv:2607.07322v1 Announce Type: cross Abstract: Automated crowd counting in Hajj video is difficult not because current models lack capacity, but because the footage violates the assumptions those m

9 Jul 2026

Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving

arXiv:2607.05399v1 Announce Type: cross Abstract: Large language model serving is increasingly limited by KV-cache growth under long-context workloads, yet existing KV-cache compression techniques are

GraphAllocBench: A Flexible Benchmark for Preference-Conditioned Multi-Objective Policy Learning

model-releasesarxiv-cs-lg

arXiv:2601.20753v4 Announce Type: replace Abstract: Preference-Conditioned Policy Learning (PCPL) in Multi-Objective Reinforcement Learning (MORL) approximates diverse Pareto-optimal solutions by cond

HoloCount: A Holistic Visual Counting Benchmark for MLLMs

arXiv:2607.06420v1 Announce Type: new Abstract: Visual counting is a fundamental pillar of multimodal intelligence, requiring a seamless integration of fine-grained grounding and spatial reasoning. Wh

PolyWorkBench: Benchmarking Multilingual Long-Horizon LLM Agents

arXiv:2607.06008v1 Announce Type: new Abstract: Large language model (LLM) agents have shown strong performance in long-horizon tasks that require planning, tool use, and interaction with external env

RuBench: A Repository-Level Agentic Coding Benchmark with Natively Authored Russian Task Specifications

arXiv:2607.06411v1 Announce Type: cross Abstract: Developers increasingly delegate real maintenance work to product-grade coding agents, and many state tasks in their native language, in the style of

UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generation

arXiv:2607.06306v1 Announce Type: cross Abstract: Large language models (LLMs) have demonstrated growing competence in web page generation. However, existing text-driven approaches rely on complex pro

A Fair Benchmarking of Deep Relational Database Learning Models

arXiv:2607.03659v1 Announce Type: cross Abstract: Relational databases (RDBs) are the primary data infrastructure in many enterprises, yet recent deep learning methods designed for RDBs have been eval

APeB: Benchmarking Personalization Ability of Large Language Model Agents

arXiv:2607.03162v1 Announce Type: new Abstract: LLM-powered agents struggle with personalization when users issue raw, underspecified queries. In this setting, agents must infer latent intent, extract

CausalGame: Benchmarking Causal Thinking of LLM Agents in Games

arXiv:2607.04293v1 Announce Type: cross Abstract: Building AI Scientist agents with Large Language Models (LLMs) has recently attracted growing attention. Since scientific discovery fundamentally reli

Applications

Deriving Benchmarking Datasets from Long-Form Recordings: Challenges and Opportunities

arXiv:2607.03201v1 Announce Type: cross Abstract: Long-form recordings (LFRs) of child-centered audio are ecologically valid sources for studying early language development, but three problems limit t

applicationsarxiv-cs-lg

EgoInertia-MI: A Multimodal Egocentric Vision and IMU Benchmark for Motor Impairment Assessment

arXiv:2607.03934v1 Announce Type: new Abstract: Motor impairments, including tremor, bradykinesia, gait abnormalities, and postural instability, are common across many neurological and movement-relate

OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers

arXiv:2607.04033v1 Announce Type: cross Abstract: Optimizer selection for large-scale model training has become a system-level design decision constrained jointly by compute, memory, tuning budget, an

PPE-Bench: A Benchmark for Evaluating MLLM Unlearning under Private-Public Entanglement

arXiv:2607.02897v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) have shown strong capabilities, but they may memorize private information from web data, raising privacy conc

Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Brain

arXiv:2510.17801v2 Announce Type: replace-cross Abstract: Building robots that can perceive, reason, and act in dynamic, unstructured environments remains a central challenge. Recent embodied systems

Towards Open-World Referring Expression Comprehension: A Benchmark with Training-free Multi-task Consistency Checker

arXiv:2605.25706v2 Announce Type: replace Abstract: Referring expression comprehension (REC) aims to localize a target object within an image based on a given expression. Although recent advances in v

TrendFact: A Benchmark Towards Hotspot Perception in Automatic Fact-Checking

arXiv:2410.15135v5 Announce Type: replace Abstract: With the surge of online misinformation, Large Language Models (LLMs) and Reasoning Large Language Models (RLMs) serving as Automatic Fact-Checking

URSA: Chemistry-Aware Benchmark for Utilitarian Retrosynthesis Assessment

arXiv:2607.04688v1 Announce Type: cross Abstract: Synthesis planning aiming to find pathways of reactions for a target molecule is one of the most important and challenging tasks in drug discovery. Re

XPlainVerse: A Million-Scale Benchmark for Explainable Deepfake Detection

arXiv:2607.03562v1 Announce Type: new Abstract: As deepfake detection models increasingly produce natural language explanations, their reasoning often remains weakly grounded in visual artifacts, limi

Beyond Pixel Diffs: Benchmarking Image Change Captioning for Web UI Visual Regression Testing

arXiv:2607.01728v1 Announce Type: cross Abstract: Visual regression testing (VRT) is a standard quality assurance step in modern software release pipelines. On every change, it re-renders user interfa

3 Jul 2026

Human Capital, Not Model Benchmarks, Predicts Hybrid Intelligence in Forecasting

arXiv:2607.02467v1 Announce Type: cross Abstract: Whether pairing people with AI helps or hurts is usually reported as a single average effect. Using a real-money prediction market (Polymarket) as an

3 Jul 2026

AV-SyncBench: Decoupled Benchmarking of Temporal and Semantic Audio-Visual Synchronization

arXiv:2607.00726v1 Announce Type: new Abstract: Audio-visual feature extraction is a fundamental component of multimodal understanding and generation tasks. However, existing evaluation protocols for

EgoSafetyBench: A Diagnostic Egocentric Video Benchmark for Evaluating Embodied VLMs as Runtime Safety Guards

arXiv:2607.00218v1 Announce Type: cross Abstract: Vision-language models (VLMs) are now proposed as runtime safety guards for embodied agents in homes and factories. A deployable guard must catch genu

EmbodimentSemantic: A Spatial Scene-Graph Dataset and Benchmark for Vision-Language Models on Embodied Manipulation Trajectories

model-releasesarxiv-cs-ro

arXiv:2607.00020v1 Announce Type: new Abstract: Spatial grounding remains a key limitation of vision-language-action (VLA) systems for robotic manipulation. While current models can recognize objects

LongVQUBench: Benchmarking Long-Term Video Quality Understanding of Vision-Language Models

arXiv:2607.01086v1 Announce Type: cross Abstract: The evaluation of long-term video quality understanding remains an open challenge for large vision-language models (LVLMs). Existing video quality ben

MemSyco-Bench: Benchmarking Sycophancy in Agent Memory

arXiv:2607.01071v1 Announce Type: cross Abstract: Memory has emerged as a cornerstone of modern LLM-based agents, supporting their evolution from single-turn assistants to long-term collaborators. How

MoHallBench: A Benchmark for Motion Hallucination in Video Large Language Models

arXiv:2607.01117v1 Announce Type: new Abstract: Video Large Language Models (VideoLLMs) have shown strong progress in video understanding, yet they still suffer from hallucinations that are inconsiste

MSQA: A Natively Sourced Multilingual and Multicultural SimpleQA Benchmark

arXiv:2607.00724v1 Announce Type: new Abstract: Multilingual fluency often invites a stronger assumption: a model that can speak a user's language must also understand the culture encoded by that lang

PHREEQC-MCQ-200: A Diagnostic Benchmark for Tool-Augmented Scientific Simulator Agents

arXiv:2607.00436v1 Announce Type: new Abstract: Large language model agents are increasingly connected to scientific software, yet it remains unclear when tool access makes scientific computation more

Arena-T2I Hard: Benchmarking and Improving Faithfulness with Dependency-Aware Checklist

arXiv:2606.31711v1 Announce Type: new Abstract: Faithfulness -- how precisely a generated image aligns with its prompt -- is increasingly central to the real-world utility of text-to-image (T2I) model

1 Jul 2026

BLUEX v2: Benchmarking LLMs on Open-Ended Questions from Brazilian University Entrance Exams

arXiv:2606.22723v2 Announce Type: replace Abstract: Although Large Language Models (LLMs) excel in many tasks, their assessment in Portuguese has received less attention, particularly for open-ended,

1 Jul 2026

Bridging Scientific Heritage: An Arabic--Russian Parallel Corpus and LLM Benchmark for Sustainable Knowledge Transfer

arXiv:2606.30943v1 Announce Type: new Abstract: Russian and Arabic are among the major languages of scientific communication. Language barriers impede the exchange of research results between these co