Search: “benchmark”

OmnilingualGAIA2: Evaluating the Multilingual Gap in Frontier AI Agents

arXiv:2608.08775v1 Announce Type: new Abstract: Agentic benchmarks aim to measure how well AI agents plan, search, execute, and recover within realistic multi-tool environments, but they are almost ex

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

P^{3}: Joint Program-and-Proof Planning for Verified Code Generation

DGX agent

arXiv:2608.09277v1 Announce Type: new Abstract: Verified code generation asks a large language model (LLM) to generate both an executable program and a machine-checkable proof that the program meets a

model-releasesarxiv-cs-ai

11 Aug 2026

PragMatch: Separating Pragmatic Incongruity from Cross-Modal Mismatch in Large Vision-Language Models

arXiv:2608.09772v1 Announce Type: new Abstract: Large Vision-Language Models (LVLMs) have demonstrated strong performance on multimodal benchmarks, yet it remains unclear whether they genuinely reason

Sci-VBench: Evaluating Knowledge- and Reasoning-Intensive Video Generation in Science Domains

arXiv:2608.09873v1 Announce Type: cross Abstract: We introduce Sci-VBench, a comprehensive benchmark for evaluating knowledge- and reasoning-intensive video generation across scientific domains. It co

Software Engineering for and with GUI Agent

arXiv:2608.09278v1 Announce Type: cross Abstract: GUI agents have advanced rapidly, producing a growing body of frameworks, benchmarks, and applications. However, this growth has outpaced the maturity

safetyarxiv-cs-ai

Unified Hallucination Fuzzing for Multimodal Large Language Models

arXiv:2608.07525v1 Announce Type: cross Abstract: Hallucination remains a persistent challenge for Multimodal Large Language Models (MLLMs), severely limiting their reliability in high-stakes applicat

VIGIL: Tackling Hallucination Detection in Image Recontextualization

arXiv:2602.14633v2 Announce Type: replace Abstract: We introduce VIGIL (Visual Inconsistency & Generative In-context Lucidity), a benchmark dataset and framework that provides a fine-grained categoriz

Ask-E: An Environment for Calibrated Question Generation

arXiv:2608.06933v1 Announce Type: cross Abstract: Today, we improve models by training and evaluating them on problems at the frontier of their abilities. Creating such problems is itself a demanding

AVCap: Reinforcing Audio-Video Joint Caption with Detail-Aware Reward

arXiv:2608.06930v1 Announce Type: new Abstract: Detailed audio-video joint captioning is essential for multimodal video understanding and generation. However, prior works are constrained by three main

Do Audio Language Models Use Paralinguistic Evidence? Counterfactual Audits for Response Evaluation

arXiv:2608.06718v1 Announce Type: new Abstract: Audio-language models (ALMs) are increasingly used as judges for speech-to-speech systems, but a judge that receives audio may not actually use paraling

Tutorials

Don't `Well, Actually' Me Unless You Know What You're Talking About: Weak Presupposition Verification Degrades General QA Performance

arXiv:2608.06539v1 Announce Type: new Abstract: False-presupposition QA (FPQA) tests LLMs on their ability to identify false presuppositions in questions and abstain or correct them rather than reinfo

tutorialsarxiv-cs-cl

ECAD: Expanding Class-Agnostic Detection Beyond Thing-Centric Objectness

arXiv:2608.06841v1 Announce Type: new Abstract: Object detection is a fundamental task in visual perception, providing structured region representations for recognition, grounding, reasoning, and inte

Geo-Spatial Concept Probing of Large Language Models: Abstraction, Compositionality, and Grounding

arXiv:2608.07353v1 Announce Type: cross Abstract: Understanding concepts is fundamental to generalization. Despite their impressive performance on a wide range of tasks, Large Language Models (LLMs) s

I Seek You in Videos: Identity-Conditioned Queries for Person-Centric Video Reasoning

arXiv:2608.07417v1 Announce Type: cross Abstract: Real-world video reasoning often involves multimodal, multi-source inputs, whereas existing video reasoning tasks typically assume a simplified video-

InsertFuse: A Unified Framework for Multi-Category Reference-Guided Image Insertion

arXiv:2608.06490v1 Announce Type: new Abstract: We present InsertFuse, a unified framework for multi-category reference-guided image insertion. Its key idea is to decouple category-specific expertise

Newton-Schulz Retraction-Based Inference Enables Hidden Quantum Markov Models to Outperform Classical HMMs

model-releasesarxiv-cs-lg

arXiv:2608.06554v1 Announce Type: new Abstract: Hidden Markov models (HMMs) are widely used probabilistic models for discrete sequential data but can be limited when hidden dynamics are complex. Hidde

Not All Problems Are Best Modeled as MILP: A DSL-Centric Framework for Flexible and Accurate Optimization Modeling

arXiv:2608.07040v1 Announce Type: new Abstract: Solving combinatorial optimization problems (COPs) requires not only efficient algorithms but also carefully crafted formulations. While recent works ha

Social World Models

arXiv:2509.00559v3 Announce Type: replace Abstract: Humans intuitively navigate social interactions by simulating unspoken dynamics and reasoning about others' perspectives, even with limited informat

A Unified Risk View of Uncertainty: Posterior Risk for Disentanglement and Evaluation Beyond Proxies

model-releasesarxiv-cs-lg

arXiv:2608.05995v1 Announce Type: new Abstract: Reliable uncertainty estimates are critical in safety-sensitive applications, where understanding the sources of predictive uncertainty is essential. Th

Robust Native Language Identification through Agentic Decomposition

arXiv:2509.16666v2 Announce Type: replace Abstract: Large language models (LLMs) often achieve high performance in native language identification (NLI) benchmarks by leveraging superficial contextual

Seeing Is Not Deciding: Can Multimodal LLMs Act as Effective CEOs?

arXiv:2608.05864v1 Announce Type: new Abstract: Large language models are increasingly applied as autonomous decision-making agents. However, in executive business decisions, existing benchmarks are l

The Low Frequency Trap: Video Language Models Fail at Simple Event Bookkeeping

arXiv:2608.06361v1 Announce Type: new Abstract: Real-world video benchmarks provide broad coverage, but their fixed clips entangle event count, rate, duration, and visual complexity, making failure mo

A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents

arXiv:2602.06052v4 Announce Type: replace-cross Abstract: Research in artificial intelligence is shifting from model innovations and benchmark scores towards problem definition and rigorous real-world

Above-ground Biomass Estimation with Geospatial Foundation Models

model-releasesarxiv-cs-lg

arXiv:2608.04792v1 Announce Type: new Abstract: Accurate estimation of Above-Ground Biomass (AGB) from satellite imagery is essential for the large-scale monitoring of carbon stocks, yet it remains a

CLIP-CC-Bench: Evaluating Paragraph-Level Video Descriptions in Video-Language Models

arXiv:2608.04302v1 Announce Type: new Abstract: Benchmarking video-language models has largely focused on short clips and single-sentence metrics, leaving open whether current systems can generate acc

safetyarxiv-cs-cv

ReGround: Restoring Visual Grounding in Multi-Step Reasoning through Self-Diagnosis and Visual Re-Examination

arXiv:2608.04385v1 Announce Type: new Abstract: Vision-Language Models (VLMs) often lose visual grounding during multi-step reasoning: as reasoning chains grow longer, later inference steps rely incre

The First EgoCross Challenge at EgoVis 2026: Cross-Domain Egocentric Video Question Answering

arXiv:2608.04589v1 Announce Type: cross Abstract: EgoCross is a cross-domain egocentric video question answering benchmark designed to evaluate whether multimodal large language models can generalize

The Yokai Learning Environment: Tracking Beliefs Over Space and Time

arXiv:2508.12480v3 Announce Type: replace Abstract: The ability to cooperate with unknown partners is a central challenge in cooperative AI and widely studied in the form of zero-shot coordination (ZS

Measurement Without Validity: The Compounding Reliability Problem in Agentic AI Evaluation

arXiv:2608.00794v2 Announce Type: replace Abstract: Agentic AI evaluation pipelines produce benchmark scores that justify deployment decisions, safety certifications, and regulatory compliance claims.

Applications

Modeling Scientific Experiment Scenes: Dataset and Model

arXiv:2608.02892v1 Announce Type: new Abstract: Scene Graph Generation (SGG) is fundamental to structured visual understanding, yet existing benchmarks focus mainly on daily life images and overlook s

applicationsarxiv-cs-cv

SocietyBench: Forecasting Counterfactual Social-World Evolution

arXiv:2608.04009v1 Announce Type: new Abstract: Large language models (LLMs), and the agents built on top of them, are now benchmarked heavily on whether they can finish a task -- fix a bug, drive a b

UrbanAgent: A Tool-Augmented Agent for Cross-System Urban Tasks

arXiv:2608.03018v1 Announce Type: new Abstract: Modern cities rely on an increasing number of digital services to operate, but residents' daily needs are still difficult to meet. Services are fragment

Where Did It Go Wrong? Process-Level Evaluation of Web Agents with Semantic State Tracking

arXiv:2606.15673v2 Announce Type: replace Abstract: Web agents act through long interaction sequences, yet existing benchmarks evaluate only terminal success, discarding all process information and of

WorldCup Arena: Prospective, Leakage-Free Evaluation of Frontier LLMs on a Live Tournament

arXiv:2608.04008v1 Announce Type: new Abstract: Benchmarks that measure the forecasting ability of large language models are almost always retrospective: the event has happened, the answer is somewher

Can You Trust the Confidence? ConfBench for Vision-Language Models on Document Extraction

arXiv:2608.01792v1 Announce Type: cross Abstract: Intelligent document processing (IDP) with vision-language models (VLMs) hinges on confidence scores trustworthy enough to route extractions between a

HopRefusalBench: Diagnosing Refusal Failures in Search-Augmented Agents for Multi-Hop Reasoning

arXiv:2608.01358v1 Announce Type: new Abstract: Search-augmented large language model agents are increasingly capable of solving knowledge-intensive tasks, but their behavior when a multi-hop question

MDWD: A Street-Level Dataset for Municipal Solid Waste Detection in Dense Urban Environments

arXiv:2608.00257v1 Announce Type: new Abstract: Automated visual monitoring of urban environments is a growing Computer Vision research area, but municipal solid waste detection remains under-represen

ScrambleToolBench: Agents Search Exhaustively Even When Their Own Map Points to the Next Step

arXiv:2608.02358v1 Announce Type: new Abstract: To operate robustly in open-world environments, autonomous agents should be able to infer the behavior of unfamiliar systems through interaction alone,

Semantic Alignment of AI Models: Concept Collapse, Checkpoint Dynamics, and Cross-Lingual Transfer

arXiv:2608.01585v1 Announce Type: new Abstract: Language model benchmarking is a difficult task. Outcome reasoning alone does not test the model's conceptualization of language and popular open-source

safetyarxiv-cs-cl

SpatioLM: Towards General Physical Spatial Intelligence in Vision-Language Models

arXiv:2608.01899v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) perform well on commonsense reasoning tasks but struggle with visual spatial reasoning. Most existing solutions introduc

Are the Financial Reasoning from LLMs Credible? A Real World Test over Long-Horizon Statements

arXiv:2607.28661v1 Announce Type: new Abstract: Do Large Language Models (LLMs) possess genuine structural reasoning, or merely rely on surface-level pattern matching? The financial domain, demanding

Harnessing the Wisdom of LLM Crowds through Complementarity-Driven Iterative Collaboration

arXiv:2607.29087v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed in enterprise settings, yet individual models remain bounded by model-specific capability limitat

Self-Play Meets Skill Evolution: Self-Evolving Search Agents that Pose, Solve, and Remember

arXiv:2607.29468v1 Announce Type: new Abstract: Self-play agents can generate training problems without questions from target benchmarks, but their curricula lack persistent state: failures affect gra

safetyarxiv-cs-ai

Step-Level Visual Grounding Faithfulness Predicts Out-of-Distribution Generalization in Long-Horizon Vision-Language Models

arXiv:2603.06828v2 Announce Type: replace-cross Abstract: We uncover a behavioral law of long-horizon vision-language models: models that maintain temporally grounded beliefs generalize better. Standa

WitCert: Sound Runtime Risk Observability and Gating for KV-Cache Quantization

arXiv:2607.28699v1 Announce Type: cross Abstract: KV-cache quantization is validated today by offline benchmark averages; a deployed system cannot tell whether compression is damaging the request it i

Baikal: Structured Search for Deep Research over Data Lakes

arXiv:2607.27726v1 Announce Type: cross Abstract: Deep research over data lakes requires an LLM agent to investigate evidence across thousands of heterogeneous tables and passages to synthesize a repo

31 Jul 2026

Beyond Borrowed Histories: Person-Aligned User Simulation for Interactive Role-Playing Evaluation

arXiv:2607.27816v1 Announce Type: new Abstract: Role-playing agents (RPAs) have become one of the most important consumer applications of large language models. Users engage in multi-turn conversation

31 Jul 2026

Beyond Similarity: Grounded Agentic Extraction and Expert-Adjudicated Evaluation of Intertextuality in Classical Chinese Histories

arXiv:2607.27595v1 Announce Type: new Abstract: Computational approaches to intertextuality have advanced from string matching to neural retrieval, yet their outputs, similarity scores and parallel-pa