Search: “benchmark”

GenTI: Benchmarking LLMs for Autonomous IDPS Rule Generation for Unseen Attacks

arXiv:2606.05844v1 Announce Type: cross Abstract: Rule-based Intrusion Detection and Prevention Systems (IDPS) offer precise attack detection as well as mitigation, however their manually crafted, sig

6 Jun 2026

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

DisasterBench: A Multimodal Benchmark for UAV-Based Disaster Response in Complex Environments

DGX agent

arXiv:2606.06217v1 Announce Type: new Abstract: When a disaster unfolds, responders must answer not only what is happening, but also why it is happening, what will happen next, and what to do now, oft

model-releasesarxiv-cs-cv

5 Jun 2026

Domain-Conditioned Safety in Frontier Computer-Using Agents: A 793-Episode Browser Benchmark, a Coding-Domain Cross-Reference, and a Reproducibility Audit of Recent Red-Teaming

arXiv:2606.05233v1 Announce Type: cross Abstract: Recent computer-using-agent (CUA) red-teaming papers report prompt-injection attack success rates (ASR) of 42-98%, but these headline numbers cluster

5 Jun 2026

PlanBench-V: A Spatial Planning Map Benchmark for Vision-Language Models

arXiv:2606.05744v1 Announce Type: new Abstract: Spatial planning maps are central to territorial governance, translating planning objectives, regulations, and spatial strategies into visual forms for

5 Jun 2026

Seeing Time: Benchmarking Chronological Reasoning and Shortcut Biases in Vision-Language Models

arXiv:2606.05702v1 Announce Type: cross Abstract: Recent advancements in Vision-Language Models (VLMs) have significantly enhanced their ability to interpret complex visual semantics, yet their capaci

5 Jun 2026

AlgoVeri: An Aligned Benchmark for Verified Code Generation on Classical Algorithms

arXiv:2602.09464v2 Announce Type: replace-cross Abstract: Vericoding refers to the generation of formally verified code from rigorous specifications. Recent AI models show promise in vericoding, but a

BBOmix: A Tabular Benchmark for Hyperparameter Optimization of Unsupervised Biological Representation Learning

model-releasesarxiv-cs-lg

arXiv:2606.05139v1 Announce Type: new Abstract: The rapid advancement of high-throughput sequencing has led to large, high-dimensional omics datasets. Deep unsupervised learning architectures, particu

CyberGym-E2E: Scalable Real-World Benchmark for AI Agents' End-to-End Cybersecurity Capabilities

arXiv:2606.04460v1 Announce Type: cross Abstract: AI has the potential to transform cybersecurity by enabling systems that can autonomously detect, analyze, and remediate software vulnerabilities. How

LifeSide: Benchmarking Agents as Lifelong Digital Companions

arXiv:2606.04660v1 Announce Type: new Abstract: Lifelong digital companions must integrate cross-session cues, continually update their understanding of users, and adapt to shifting privacy boundaries

MemoryDocDataSet: A Benchmark for Joint Conversational Memory and Long Document Reasoning

arXiv:2606.04442v1 Announce Type: cross Abstract: AI systems increasingly need to combine two demanding capabilities: navigating multi-session conversation history and performing deep reading comprehe

MineXplore: An Open-Source Reinforcement Learning Exploration Benchmark for GNSS-Denied Underground Environment

model-releasesarxiv-cs-ro

arXiv:2606.04569v1 Announce Type: new Abstract: Underground mines present extreme conditions for autonomous robot navigation: GPS is denied, lighting is degraded, and tunnel topology is loop-rich and

Plan, Watch, Recover: A Benchmark and Architectures for Proactive Procedural Assistance

arXiv:2606.04970v1 Announce Type: cross Abstract: We envision a proactive multi-modal assistant system which gives users real-time step-by-step guidance on a procedural task, autonomously deciding ext

PoliticsBench: Benchmarking Political Values in Large Language Models with Multi-Turn Roleplay

arXiv:2603.23841v2 Announce Type: replace-cross Abstract: While Large Language Models (LLMs) are increasingly used as primary sources of information, their potential for political bias may impact thei

WETBench: A Benchmark for Detecting Task-Specific Machine-Generated Text on Wikipedia

arXiv:2507.03373v2 Announce Type: replace Abstract: Given Wikipedia's role as a trusted source of high-quality, reliable content, concerns are growing about the proliferation of low-quality machine-ge

AnyAudio-Judge: A Dynamic Rubric-Based Benchmark and Evaluator for Audio Instruction Following

arXiv:2606.03116v1 Announce Type: cross Abstract: The rapid advancement of instruction-guided audio generation has highlighted the critical need for robust alignment evaluation. Current automated eval

Characterizing Detectability in 3DGS Poisoning: A Stage-wise Benchmark

arXiv:2606.03499v1 Announce Type: new Abstract: 3D Gaussian Splatting (3DGS) has rapidly emerged as a leading representation for real-time novel view synthesis, but recent work shows it is vulnerable

CoMPAS3D: A Dataset and Benchmark for Interactive Motion

arXiv:2507.19684v2 Announce Type: replace-cross Abstract: Socially interactive humanoid robots must engage with humans through their bodies, adapting in real time to a partner's movement, intent, and

Cryo-Bench: Benchmarking Foundation Models for Cryosphere Applications

arXiv:2603.01576v3 Announce Type: replace Abstract: Geo-Foundation Models (GFMs) have been evaluated across diverse Earth observation task including multiple domains and have demonstrated strong poten

DeskCraft: Benchmarking Desktop Agents on Professional Workflows and Human-in-the-Loop Collaboration

arXiv:2606.03103v1 Announce Type: new Abstract: Real-world professional desktop workflows in specialized creative and engineering software unfold over long horizons and often require human-in-the-loop

Diagnosing Knowledge Gaps in LLM Tool Use: An Agentic Benchmark for Novel API Acquisition

arXiv:2606.03657v1 Announce Type: new Abstract: Large language models for code generation often need to use APIs that are absent from their pretraining data. This requires more than recalling a functi

Gate AI: LLM Security Benchmark Evaluation Methodology and Results

model-releasesarxiv-cs-lg

arXiv:2606.02959v1 Announce Type: new Abstract: Published evaluations of prompt-injection and jailbreak detectors for Large Language Models often suffer from two systematic weaknesses: per-dataset thr

MedCUA-Bench: A Screenshot-Only Benchmark for Clinical Computer-Use Agents

arXiv:2606.03203v1 Announce Type: new Abstract: Computer-use agents could automate repetitive screen-based clinical work, but their reliability in medical graphical user interfaces remains largely unv

OVO-S-Bench: A Hierarchical Benchmark for Streaming Spatial Intelligence in Multimodal LLMs

arXiv:2606.03890v1 Announce Type: new Abstract: Multimodal agents in robotics, AR, and autonomous driving must reason about places and layouts from continuous egocentric streams, often using evidence

SynCred-Bench: Benchmarking Synthetic Credibility in AI-Generated Visual Misinformation

arXiv:2606.03348v1 Announce Type: cross Abstract: Recent generative models can now produce visual artifacts with realistic embedded text and layouts, creating a new misinformation threat: synthetic cr

TeX-1500: A Paired Real-World LWIR Hyperspectral Dataset and Benchmark for Temperature-Emissivity-Texture Decomposition

arXiv:2606.03806v1 Announce Type: new Abstract: Temperature-emissivity-texture (TeX) decomposition seeks to recover object heat state, material spectral response, and visible-like geometric texture fr

VidMsg: A Benchmark for Implicit Message Inference in Short Videos

arXiv:2606.03635v1 Announce Type: cross Abstract: Understanding short online videos involves more than identifying visible objects and actions; video makers often include an underlying message or purp

VLA-Arena: An Open-Source Framework for Benchmarking Vision-Language-Action Models

arXiv:2512.22539v2 Announce Type: replace-cross Abstract: While Vision-Language-Action models (VLAs) are rapidly advancing towards generalist robot policies, it remains difficult to quantitatively und

Whose Name Comes Up? II: Benchmarking and Intervention-Based Auditing of LLM-Based Scholar Recommendation

arXiv:2602.08873v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are now used for academic expert recommendation. Existing audits typically evaluate such recommendations in isola

An Open-Source Benchmark and Baseline for Multi-temporal Referring Segmentation

arXiv:2606.00987v1 Announce Type: cross Abstract: Large Vision-Language Models (LVLMs) have shown strong visual understanding and language-guided grounding abilities, yet their capacity for multi-temp

Benchmarking Large Language Models for Cryptanalysis and Side-Channel Vulnerabilities

arXiv:2505.24621v3 Announce Type: replace Abstract: Recent advancements in large language models (LLMs) have transformed natural language understanding and generation, leading to extensive benchmarkin

Benchmarking LLM-as-a-Judge for Long-Form Output Evaluation

arXiv:2606.01629v1 Announce Type: new Abstract: As large language models (LLMs) are increasingly used for long-form generation, reliably evaluating long-form outputs has become a critical challenge. L

Can LLMs Reason Structurally? Benchmarking via the Lens of Data Structures

arXiv:2505.24069v4 Announce Type: replace-cross Abstract: Large language models (LLMs) are deployed on increasingly complex tasks that require multi-step decision-making. Understanding their algorithm

ChartArena: Benchmarking Chart Parsing across Languages, Scenarios, and Formats

arXiv:2606.01348v1 Announce Type: new Abstract: Charts are a primary medium for conveying quantitative and relational information, yet systematically evaluating chart parsing models remains difficult.

CoCoVideo: The High-Quality Commercial-Model-Based Contrastive Benchmark for AI-Generated Video Detection

arXiv:2606.00101v1 Announce Type: cross Abstract: With the rapid advancement of artificial intelligence generated content (AIGC) technologies, video forgery has become increasingly prevalent, posing n

Do Gender Cues Affect LLM Value Trade-offs? Evidence from a Controlled Decision Benchmark

arXiv:2606.02214v1 Announce Type: new Abstract: Large language models are increasingly used in value-sensitive decision settings, where irrelevant demographic cues should not alter judgments. We const

Does Compression Preserve Uncertainty? A Unified Benchmark for Quantized and Sparse LLMs via Conformal Prediction

arXiv:2606.01850v1 Announce Type: new Abstract: Model compression techniques such as quantization and pruning are widely used to reduce the deployment cost of large language models (LLMs), with existi

Ego-METAS: Egocentric online Multimodal Energy-efficient Temporal Action Segmentation benchmark

arXiv:2606.02246v1 Announce Type: new Abstract: To operate in the physical world, embodied agents must perceive their environment in an 'always-on' fashion, selectively accessing the most informative

Evaluating Interactive Reasoning in Large Language Models: A Hierarchical Benchmark with Executable Games

arXiv:2606.00103v1 Announce Type: new Abstract: We introduce a multi-turn interactive framework for reasoning evaluation that treats reasoning as active evidence acquisition and belief updating. Where

InsightVQA: High-Dimensional Emotion-Cognitive Visual Question Answering Benchmark

arXiv:2606.02171v1 Announce Type: new Abstract: Visual emotion understanding requires models not only to recognize emotional states, but also to why they arise and perform higher-level cognitive reaso

K-BrowseComp: A Web Browsing Agent Benchmark Grounded in Korean Contexts

arXiv:2606.02404v1 Announce Type: new Abstract: Frontier model evaluations are shifting from foundational capabilities (e.g., instruction following and reasoning) toward compositional, agentic ones, b

MMDG-Bench: A Benchmark for Multimodal Domain Generalization

arXiv:2606.00891v1 Announce Type: new Abstract: Multi-modal Domain Generalization (MMDG) seeks to leverage complementary modalities to enhance model robustness on unseen domains. Despite extensive pro

Agents

OpenHospital: A Thing-in-itself Arena for Evolving and Benchmarking LLM-based Collective Intelligence

arXiv:2603.14771v3 Announce Type: replace Abstract: Large Language Model (LLM)-based Collective Intelligence (CI) presents a promising approach to overcoming the data wall and continuously boosting th

agentsarxiv-cs-ai

PaSBench-Video: A Streaming Video Benchmark for Proactive Safety Warning

arXiv:2606.02443v1 Announce Type: cross Abstract: Between the first visible sign of danger and the moment an accident occurs, there is often a window where intervention remains possible. Video-capable

PolySpeech-100: A Large-Scale Benchmark for Speech Understanding Across 100+ Languages and Dialects

arXiv:2606.01016v1 Announce Type: cross Abstract: While End-to-End (E2E) Speech-Large Language Models (Speech-LLMs) are rapidly evolving, their evaluation methodologies remain limited to the era of si

Local Ai

Position: Current Benchmarking Hinders Real Progress in Deep Learning for Time Series Forecasting

arXiv:2512.22702v2 Announce Type: replace Abstract: Deep learning models have grown popular in time series applications. However, the large quantity of newly proposed architectures and the often contr

local-aiarxiv-cs-lg

RoboStressBench: Benchmarking VLM Robustness to Physical Visual Stress in Embodied Scenes

arXiv:2606.00828v1 Announce Type: new Abstract: Vision-Language Models (VLMs) have shown strong visual understanding and are increasingly deployed in embodied AI systems, where reliable perception und

SMH-Bench: Benchmarking LLM Agents for Environment-Grounded Reasoning and Action in Smart Homes

arXiv:2606.01912v1 Announce Type: new Abstract: Smart homes are evolving toward complex state-dependent living environments, requiring Large Language Models (LLMs) to reason over user intent, preferen

Agents

Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends

arXiv:2606.01164v1 Announce Type: new Abstract: With rapid development of large language models and diffusion-based content generation, world modeling has attracted increasing research attention, bene

agentsarxiv-cs-cv