Search: “benchmark”

AHA-Memes: A Fine-Grained Multimodal Benchmark for Understanding Hate in Arabic Memes

arXiv:2607.27393v1 Announce Type: new Abstract: Hateful memes are a growing form of multimodal online harm, where hostile intent is often conveyed through the joint interpretation of images, text, cul

31 Jul 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

MMHBench: A Multi-Perspective Benchmark for Mental Health Understanding in Long-Form Videos

DGX agent

arXiv:2607.27895v1 Announce Type: cross Abstract: Mental health understanding in long-form videos requires nuanced reasoning over observable behavior, interpersonal context, and latent psychological s

model-releasesarxiv-cs-cv

31 Jul 2026

PlatformBid: An Auto-Bidding Benchmark from a Unified Advertising Platform's Perspective

arXiv:2607.27265v1 Announce Type: new Abstract: Real-time bidding is central to computational advertising, comprising three elements: Supply Side Platform (SSP) selling ad impressions, Demand Side Pla

31 Jul 2026

When Synthetic Users Fail: A Cross-Domain Benchmark of LLM-Simulated Human Survey Responses

arXiv:2607.26348v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used as synthetic users, stand-ins for human respondents whose simulated answers feed product, policy, and

30 Jul 2026

HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following

arXiv:2607.25398v1 Announce Type: new Abstract: Language-model agents are increasingly deployed under standing instructions: a system prompt, a policy file, or a skills document is placed in context,

29 Jul 2026

RefBench-PRO: Perceptual and Reasoning Oriented Benchmark for Referring Expression Comprehension

arXiv:2512.06276v3 Announce Type: replace-cross Abstract: Referring Expression Comprehension (REC) is a vision-language task that localizes a specific image region based on a textual description. Exis

29 Jul 2026

WorkSurface-Bench: Benchmarking Enterprise Agents on Multi-Surface Knowledge Routing

arXiv:2607.25765v1 Announce Type: new Abstract: Enterprise agents often need to integrate heterogeneous knowledge sources: documents for narrative facts, tables for computation, and dependency graphs

29 Jul 2026

A Controlled Visual-Backbone Benchmark for Multimodal Short-Term Solar Irradiance Forecasting

arXiv:2607.23633v1 Announce Type: cross Abstract: Sky-image irradiance studies often compare forecasting systems in which the image encoder, temporal model, fusion block, target definition, and traini

DriveDNA: A Large-Scale Multimodal Naturalistic Driving Dataset and Benchmark for Driving Style Identification

arXiv:2607.23822v1 Announce Type: new Abstract: Driving style captures stable, driver-specific patterns in how a vehicle is driven. In naturalistic data, however, this signal is hard to isolate becaus

Subject-Level Heterogeneity in EEG Motor Imagery Decoding: A Large-Scale Benchmark and Portfolio-Based Reduction of the Search Space

arXiv:2607.22778v1 Announce Type: cross Abstract: Robust EEG motor imagery decoding remains limited by strong inter-individual variability, making it difficult to identify pipelines that generalize ac

When Less Is More: A Controlled Benchmark of Lightweight CNNs for Satellite Land-Cover Segmentation on DeepGlobe

arXiv:2607.23024v1 Announce Type: new Abstract: High-resolution satellite imagery is the backbone of good land-cover classification, and without that, environmental monitoring, urban planning, and sus

DBA-Bench: A Production-Fidelity Benchmark for LLM-Based Database Operations Agents

arXiv:2607.22165v1 Announce Type: cross Abstract: LLM-based database agents show promise, but differing task scopes, testbeds, and metrics hinder comparison. We identify four gaps between evaluation a

27 Jul 2026

ReCowGnition: A Realistic Biometric Benchmark for Cow Face Recognition

arXiv:2607.22071v1 Announce Type: new Abstract: With the development of precision livestock farming and the advances in computer vision, visual animal biometrics has gained attention. Using biometric

27 Jul 2026

AISE-Bench: A Full-Cycle Curated Benchmark for Information Seeking on Academic Knowledge Graphs

arXiv:2607.20498v1 Announce Type: new Abstract: Large language models (LLMs) augmented with tools are emerging as autonomous agents capable of using Web engine, APIs, and code to solve complex, long-h

CANN Bench: Benchmarking Agent Generated Kernels against Real NPU and Algorithmic Limits

arXiv:2607.20518v1 Announce Type: new Abstract: AI agents are now capable of writing, compiling, and iteratively optimizing low-level operator kernels on different hardware platforms. Existing benchma

PersonaTrail: Benchmarking Personalized Web Agents through Browsing Trails

arXiv:2607.20482v1 Announce Type: new Abstract: Recent advances in large language models have enabled web agents to autonomously execute complex tasks. In practice, users frequently provide underspeci

QuantiBias: Benchmarking Quantization-Induced Bias in LLMs

arXiv:2607.21063v1 Announce Type: new Abstract: Almost every large language model that reaches a broad audience is quantized: trained in full precision, then compressed for efficiency. This step is as

The RealDefocus Benchmark for Defocus Deblurring

arXiv:2607.21078v1 Announce Type: new Abstract: Single-Image Defocus Deblurring (SIDD) aims to recover an all-in-focus image from a single defocused observation, but rigorous and reproducible evaluati

VoLN: Vision-Only Long-Horizon Navigation---Paradigm, Benchmark, and Method

arXiv:2607.21400v1 Announce Type: cross Abstract: Vision-and-Language Navigation (VLN) enables embodied agents to follow natural-language instructions. However, route-level instructions commonly encod

WildTrace: Benchmarking Natural Evidence Trails in Long-Context Reasoning

arXiv:2607.09328v2 Announce Type: replace-cross Abstract: Answering complex questions over long documents frequently requires integrating evidence that the source itself disperses naturally across dis

CrackedPDFs: A Controlled Benchmark for Hidden Prompt Injection in PDFs

arXiv:2607.19396v1 Announce Type: new Abstract: Document-based LLM systems often flatten a PDF before guardrails inspect it. That step can discard evidence that an instruction was never visible to the

CruiseBench: A Real-Flight-Aligned N-CMAPSS Benchmark for Engine RUL Prediction

arXiv:2607.19380v1 Announce Type: new Abstract: Remaining useful life (RUL) prediction estimates how long an engine can continue safe operation and is central to maintenance planning. N-CMAPSS extends

ExpertVerse: A General-Purpose Benchmark for Expert-Level Reasoning in Knowledge-Intensive Visual Synthesis

arXiv:2607.19341v1 Announce Type: new Abstract: Recent advances in multimodal generative models have enabled instruction-based image generation to move beyond semantic manipulation to knowledge-driven

Learning Explicit Physical Parameter Control and Benchmarking for Video Generation

arXiv:2607.18924v1 Announce Type: new Abstract: Recent advances in image-to-video generation have improved visual realism, making physically grounded and controllable dynamics an important step toward

NavVerse: Benchmarking Indoor-to-Outdoor Embodied Navigation in Continuous Robot Simulation

model-releasesarxiv-cs-ro

arXiv:2607.19695v1 Announce Type: new Abstract: Robots deployed in delivery, campus, and emergency-response settings often need to navigate from buildings to streets within a single continuous episode

Open-Vocabulary Gaze Object Prediction: Benchmark and Method

arXiv:2607.18827v1 Announce Type: new Abstract: Gaze Object Prediction (GOP) aims to localize and recognize the objects humans attend to, a task crucial for understanding human-centric interactions. H

OpenSkillRisk: Benchmarking Agent Safety When Using Real-World Risky Third-Party Skills

arXiv:2607.20121v1 Announce Type: new Abstract: LLM-based agents leverage third-party skills to extend their capabilities in open-world scenarios. However, third-party skills can introduce extra secur

PhenSPINE: A Standardized Benchmark for Spine Pathology Diagnosis

arXiv:2607.19696v1 Announce Type: cross Abstract: The accurate diagnosis of spinal pathologies depends heavily on radiological interpretation, yet automated systems are hindered by the lack of diverse

DevicesWorld: Benchmarking Cross-Device Agents in Heterogeneous Environments

arXiv:2607.13465v1 Announce Type: cross Abstract: LLM-based agents have rapidly improved at operating individual digital environments such as mobile applications, desktop systems, and smart homes. How

16 Jul 2026

nuTruck: Benchmarking Autonomous Driving Planning for Distributed Electric-drive Trucks

model-releasesarxiv-cs-ro

arXiv:2607.13704v1 Announce Type: new Abstract: The dominance of traditional rule-based methods in autonomous driving has gradually been replaced by learning-based approaches. While learning-based pla

16 Jul 2026

RAGthoven at SemEval-2026 Task 1: A Multi-Stage Pipeline Walks Into a Benchmark and Barely Clears the Bar

arXiv:2607.13189v1 Announce Type: cross Abstract: We present RAGthoven, our system for SemEval-2026 Task 1 (MWAHAHA), Subtask A (multilingual constrained humor generation in English, Spanish, and Chin

16 Jul 2026

Safety

Operationalising Multi-Dimensional Evaluation for Conversational Agents: A Scalable, Governed Pipeline with Selective Re-evaluation and Model Benchmarking

arXiv:2607.12085v1 Announce Type: new Abstract: Evaluating retail conversational agents requires methods beyond lexical-overlap metrics to assess intent alignment, factuality, helpfulness, clarity, to

safetyarxiv-cs-ai

15 Jul 2026

HumanForge: A Human-Centric Deepfake Video Benchmark with Multi-Agent Forgery Rationales

arXiv:2607.08705v1 Announce Type: new Abstract: Rapid advancements in video diffusion models and temporal editing tools have enabled the generation of highly realistic human-centric videos, posing unp

10 Jul 2026

UniRef-UAV: A Multimodal Benchmark for Universal Referring in UAV Imagery

arXiv:2607.08267v1 Announce Type: new Abstract: Unmanned aerial vehicles (UAVs) increasingly rely on visual grounding capabilities to localize task-relevant targets from diverse instructions in comple

10 Jul 2026

Does AI Understand Imaging? A Systematic Benchmark of Agentic AI for Computational Imaging Tasks

arXiv:2607.07189v1 Announce Type: new Abstract: Vision-language models (VLMs) and agentic AI have shown strong performance on semantic visual tasks, but it remains unclear whether they can handle the

9 Jul 2026

Generalist Vision-Language Models for Fast Radio Burst detection: a zero-shot benchmark against a specialized detector

arXiv:2607.07382v1 Announce Type: new Abstract: Fast Radio Bursts (FRBs) are millisecond-duration radio transients whose automated detection increasingly relies on highly specialized deep learning mod

9 Jul 2026

HAJJv2-CrowdCount: Zero-Shot Benchmark for Dense Crowd Counting

arXiv:2607.07322v1 Announce Type: cross Abstract: Automated crowd counting in Hajj video is difficult not because current models lack capacity, but because the footage violates the assumptions those m

9 Jul 2026

Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving

arXiv:2607.05399v1 Announce Type: cross Abstract: Large language model serving is increasingly limited by KV-cache growth under long-context workloads, yet existing KV-cache compression techniques are

GraphAllocBench: A Flexible Benchmark for Preference-Conditioned Multi-Objective Policy Learning

arXiv:2601.20753v4 Announce Type: replace Abstract: Preference-Conditioned Policy Learning (PCPL) in Multi-Objective Reinforcement Learning (MORL) approximates diverse Pareto-optimal solutions by cond

HoloCount: A Holistic Visual Counting Benchmark for MLLMs

arXiv:2607.06420v1 Announce Type: new Abstract: Visual counting is a fundamental pillar of multimodal intelligence, requiring a seamless integration of fine-grained grounding and spatial reasoning. Wh

PolyWorkBench: Benchmarking Multilingual Long-Horizon LLM Agents

arXiv:2607.06008v1 Announce Type: new Abstract: Large language model (LLM) agents have shown strong performance in long-horizon tasks that require planning, tool use, and interaction with external env

RuBench: A Repository-Level Agentic Coding Benchmark with Natively Authored Russian Task Specifications

arXiv:2607.06411v1 Announce Type: cross Abstract: Developers increasingly delegate real maintenance work to product-grade coding agents, and many state tasks in their native language, in the style of

UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generation

arXiv:2607.06306v1 Announce Type: cross Abstract: Large language models (LLMs) have demonstrated growing competence in web page generation. However, existing text-driven approaches rely on complex pro

A Fair Benchmarking of Deep Relational Database Learning Models

arXiv:2607.03659v1 Announce Type: cross Abstract: Relational databases (RDBs) are the primary data infrastructure in many enterprises, yet recent deep learning methods designed for RDBs have been eval

APeB: Benchmarking Personalization Ability of Large Language Model Agents

arXiv:2607.03162v1 Announce Type: new Abstract: LLM-powered agents struggle with personalization when users issue raw, underspecified queries. In this setting, agents must infer latent intent, extract

CausalGame: Benchmarking Causal Thinking of LLM Agents in Games

arXiv:2607.04293v1 Announce Type: cross Abstract: Building AI Scientist agents with Large Language Models (LLMs) has recently attracted growing attention. Since scientific discovery fundamentally reli

Applications

Deriving Benchmarking Datasets from Long-Form Recordings: Challenges and Opportunities

arXiv:2607.03201v1 Announce Type: cross Abstract: Long-form recordings (LFRs) of child-centered audio are ecologically valid sources for studying early language development, but three problems limit t

applicationsarxiv-cs-lg

EgoInertia-MI: A Multimodal Egocentric Vision and IMU Benchmark for Motor Impairment Assessment

arXiv:2607.03934v1 Announce Type: new Abstract: Motor impairments, including tremor, bradykinesia, gait abnormalities, and postural instability, are common across many neurological and movement-relate