Search: “benchmark”

MR-LiDAR: A Multi-Resolution Roadside LiDAR Benchmark for Perception Diagnostics and Deployment Guidance

model-releasesarxiv-cs-ro

arXiv:2605.24777v1 Announce Type: new Abstract: LiDAR model selection is a critical issue in roadside sensing systems, as it directly determines both perception capability and deployment cost. However

26 May 2026

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Rethinking Weak Supervision in Anomaly Detection: A Comprehensive Benchmark

DGX agent

arXiv:2605.26068v1 Announce Type: cross Abstract: Weakly supervised anomaly detection (WSAD) has developed in three primary directions: incomplete, inexact, and inaccurate supervision. However, these

model-releasesarxiv-cs-ai

26 May 2026

TS-Skill: A Benchmark for Evaluating Analytical Skills in Time-Series Question Answering

arXiv:2605.24703v1 Announce Type: cross Abstract: Large language models (LLMs) and time-series language models (TSLMs) are increasingly applied to time-series question answering (TSQA). Unlike text-on

26 May 2026

WorldGUI: An Interactive Benchmark for Desktop GUI Automation from Any Starting Point

arXiv:2502.08047v5 Announce Type: replace Abstract: Recent progress in GUI agents has substantially improved visual grounding, yet robust planning remains challenging, particularly when the environmen

26 May 2026

Are Frontier LLMs Ready for Cybersecurity? Evidence for Vertical Foundation Models from Dual-Mode Vulnerability Benchmarks

arXiv:2605.23243v1 Announce Type: cross Abstract: We evaluate whether frontier LLMs are ready for cybersecurity through a dual-mode benchmark: white-box function-level vulnerability detection (VulnLLM

Benchmarking and Enhancing VLM for Compressed Image Understanding

arXiv:2512.20901v2 Announce Type: replace Abstract: With the rapid development of Vision-Language Models (VLMs) and the growing demand for their applications, efficient compression of the image inputs

Benchmarking Google Embeddings 2 against Open-Source Models for Multilingual Dense Retrieval and RAG Systems

arXiv:2605.23618v1 Announce Type: new Abstract: We benchmark Google Embeddings (GE2), a Vertex-AI-hosted bi-encoder with 2,048-token context and explicit task-type conditioning, against five open-sour

ChartFI: Benchmarking Faithfulness and Insightfulness of Chart Descriptions from Multimodal Large Language Models

arXiv:2605.23694v1 Announce Type: new Abstract: Chart descriptions are essential for accessibility, cross-modal retrieval, and assisting readers in extracting insights from complex visualizations. As

DRIVESPATIAL: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Driving

arXiv:2605.23176v1 Announce Type: new Abstract: Spatiotemporal intelligence in autonomous driving (AD) requires an agent to integrate multi-view observations into a coherent scene representation, main

SciHorizon-GENE: Benchmarking LLM for Life Sciences Inference from Gene Knowledge to Functional Understanding

arXiv:2601.12805v3 Announce Type: replace-cross Abstract: Large language models (LLMs) have shown growing promise in biomedical research, particularly for knowledge-driven interpretation tasks. Howeve

COCOTree: A Dataset and Benchmark for Open Tree-Structured Visual Decomposition

arXiv:2605.22068v1 Announce Type: new Abstract: We formalize and enable the task of open tree decomposition, which segments an image into hierarchical trees of visual components with unconstrained gra

Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements

arXiv:2605.22079v1 Announce Type: new Abstract: Large language models (LLMs) are widely used to generate structured outputs such as JSON, SQL, and code, yet public resources remain limited for evaluat

JMed48k: A Multi-Profession Japanese Medical Licensing Benchmark for Vision-Language Model Evaluation

arXiv:2605.22080v1 Announce Type: new Abstract: We introduce JMed48k, a multi-profession Japanese healthcare licensing benchmark for evaluating vision-language models. Built from official PDF material

Rethinking Noise-Robust Training for Frozen Vision Foundation Models: A Cross-Dataset Benchmark with a Case Study of Small-Loss Failure

arXiv:2605.22591v1 Announce Type: new Abstract: Frozen Vision Foundation Models (VFMs) with lightweight classification heads are increasingly used in medical imaging because they offer efficient and r

Seizure-Semiology-Suite (S3): A Clinically Multimodal Dataset, Benchmark, and Models for Seizure Semiology Understanding

arXiv:2605.21852v1 Announce Type: new Abstract: While Multimodal Large Language Models (MLLMs) have demonstrated remarkable proficiency in general video understanding, their capacity to interpret invo

SURGE: An Event-Centric Social Media Sentiment Time Series Benchmark with Interaction Structure

arXiv:2605.21198v1 Announce Type: cross Abstract: Public events on social media generate large volumes of discussion whose collective dynamics carry direct value for opinion forecasting and crisis res

VEELA: A Clinically-Constrained Benchmark for Liver Vessel Segmentation in Computed Tomography Angiography

arXiv:2605.22357v1 Announce Type: new Abstract: Accurate segmentation of hepatic and portal vessels in contrast-enhanced computed tomography angiography (CTA) remains challenging due to complex vascul

When Cases Get Rare: A Retrieval Benchmark for Off-Guideline Clinical Question Answering

arXiv:2605.21807v1 Announce Type: new Abstract: Across medical specialties, clinical practice is anchored in evidence-based guidelines that codify best studied diagnostic and treatment pathways. These

ShapeBench: A Scalable Benchmark and Diagnostic Suite for Standardized Evaluation in Aerodynamic Shape Optimization

model-releasesarxiv-cs-lg

arXiv:2605.20763v1 Announce Type: new Abstract: Rapid progress in aerodynamic shape optimization (ASO) has outpaced currently-available standardized evaluation frameworks. Fair comparison requires a u

21 May 2026

TextSculptor: Training and Benchmarking Scene Text Editing

arXiv:2605.21090v1 Announce Type: new Abstract: Recent advances in Multimodal Large Language Models (MLLMs) and diffusion-based generative models have substantially improved prompt-driven image editin

21 May 2026

WCXB: A Multi-Type Web Content Extraction Benchmark

arXiv:2605.21097v1 Announce Type: new Abstract: Web content extraction - isolating a page's main content from surrounding boilerplate - is a prerequisite for search indexing, retrieval-augmented gener

21 May 2026

Benchmarking Commercial ASR Systems on Code-Switching Speech: Arabic, Persian, and German

arXiv:2605.19069v1 Announce Type: cross Abstract: Code-switching -- the natural alternation between two languages within a single utterance -- represents one of the most challenging and under-studied

BLINKG: A Benchmark for LLM-Integrated Knowledge Graph Generation

arXiv:2605.19518v1 Announce Type: new Abstract: Generating Knowledge Graphs (KGs) remains one of the most time-consuming and labor-intensive tasks for knowledge engineers, as they need to identify sem

EgoCoT-Bench: Benchmarking Grounded and Verifiable Operation-Centric Chain of Thought Reasoning for MLLMs

arXiv:2605.19559v1 Announce Type: cross Abstract: The rapid development of Multimodal Large Language Models (MLLMs) has led to growing interest in egocentric video understanding, specifically the abil

MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMs

arXiv:2511.14159v2 Announce Type: replace Abstract: Evaluating the robustness of Large Vision-Language Models (LVLMs) is essential for their continued development and responsible deployment in real-wo

PRISM: A Benchmark for Programmatic Spatial-Temporal Reasoning

arXiv:2605.19382v1 Announce Type: new Abstract: Programmatic video generation through code offers geometric precision and temporal coherence beyond pixel-level diffusion models, yet rigorously evaluat

RoboJailBench: Benchmarking Adversarial Attacks and Defenses in Embodied Robotic Agents

model-releasesarxiv-cs-ro

arXiv:2605.19328v1 Announce Type: cross Abstract: Recent advances in Vision-Language Models (VLMs) facilitate a new class of embodied AI systems, where these models are integrated into physical platfo

Synthetic Data Generation for Brain-Computer Interfaces: Overview, Benchmarking, and Future Directions

arXiv:2603.12296v2 Announce Type: replace-cross Abstract: Deep learning has achieved transformative performance across diverse domains, largely driven by large-scale and high-quality training data. In

The World Won't Stay Still: Programmable Evolution for Agent Benchmarks

arXiv:2603.05910v2 Announce Type: replace Abstract: LLM-powered tool-calling agents fulfill user requests by interacting with environments, querying data, and invoking tools in a multi-turn process. Y

Trajectory Planning and Control near the Limits: an Open Experimental Benchmark on the RoboRacer Platform

model-releasesarxiv-cs-ro

arXiv:2605.19881v1 Announce Type: new Abstract: We present a modular framework to benchmark new and existing methods for trajectory planning and control in high-acceleration maneuvers that push autono

Local Ai

What hardware actually powers open-source AI? Not benchmarks. Not vendor marketing. Real-world community usage. We’re launching @huggingface…

local-aigeorgi-gerganov--x

What hardware actually powers open-source AI? Not benchmarks. Not vendor marketing. Real-world community usage. We’re launching @huggingface Hardware: → trending GPUs & CPUs → VRAM distribution → infe

AgentKernelArena: Generalization-Aware Benchmarking of GPU Kernel Optimization Agents

arXiv:2605.16819v1 Announce Type: cross Abstract: GPU kernel optimization is increasingly critical for efficient deep learning systems, but writing high-performance kernels still requires substantial

Benchmarking inference at scale: coding agents

model-releasestogether-ai-blog

This article presents benchmarking results for AI coding agents evaluated at scale, likely comparing performance metrics such as code generation accuracy, execution success rates, and inference effici

Benchmarking Mythos-Linked Bug Rediscovery

arXiv:2605.17416v1 Announce Type: cross Abstract: Anthropic's April 2026 Mythos materials combine benchmark claims with concrete bug-finding stories across OpenBSD, FreeBSD, Linux, FFmpeg, and browser

BoLT: A Benchmark to Democratize Black-box Optimization Research for Expensive LLM Tasks

arXiv:2605.17000v1 Announce Type: cross Abstract: Optimization of LLM training and inference configurations, such as hyperparameters, data mixtures, and prompts, is critical to performance, but it is

Can LLMs Think Like Consumers? Benchmarking Crowd-Level Reaction Reconstruction with ConsumerSimBench

arXiv:2605.17079v1 Announce Type: cross Abstract: LLMs are increasingly used as ``digital consumers'' to simulate public opinion, pre-test marketing decisions, and anticipate audience response. Howeve

Agents

Curious finding while creating evals and benchmarks for long-horizon (100+ turn) agents While it’s generally thought that a direct swap to o…

Curious finding while creating evals and benchmarks for long-horizon (100+ turn) agents While it’s generally thought that a direct swap to open source models can bring immediate cost savings, that’s n

agentsharrison-chase--x

DeTrack: A Benchmark and Altitude-Aware Dual World Model for Drone-embodied Tracking

arXiv:2605.17451v1 Announce Type: new Abstract: Aerial object tracking has broad applications in public safety, emergency rescue, wildlife monitoring, and related fields. However, existing aerial trac

DisasterVQA: A Visual Question Answering Benchmark Dataset for Disaster Scenes

arXiv:2601.13839v2 Announce Type: replace Abstract: Social media imagery provides a low-latency source of situational information during natural and human-induced disasters, enabling rapid damage asse

Evaluating Deep Research Agents on Expert Consulting Work: A Benchmark with Verifiers, Rubrics, and Cognitive Traps

arXiv:2605.17554v1 Announce Type: new Abstract: Frontier deep research agents (DRAs) plan a research task, synthesize across documents, and return a structured deliverable on demand. They are being de

EvilGenie: A Reward Hacking Benchmark

model-releasesarxiv-cs-lg

arXiv:2511.21654v2 Announce Type: replace Abstract: We introduce EvilGenie, a benchmark for reward hacking in programming settings. We source problems from LiveCodeBench and create an environment in w

FinAuditing: A Financial Taxonomy-Structured Multi-Document Benchmark for Evaluating LLMs

arXiv:2510.08886v3 Announce Type: replace Abstract: Going beyond simple text processing, financial auditing requires detecting semantic, structural, and numerical inconsistencies across large-scale di

FinTagging: Benchmarking LLMs for Extracting and Structuring Financial Information

arXiv:2505.20650v5 Announce Type: replace-cross Abstract: Accurate interpretation of numerical data in financial reports is critical for markets and regulators. Although XBRL (eXtensible Business Repo

LEAF: A Living Benchmark for Event-Augmented Forecasting

arXiv:2605.16358v1 Announce Type: cross Abstract: Large Language Models (LLMs) are increasingly applied to forecasting. To evaluate this capability while mitigating pre-training data contamination, se

PluRule: A Benchmark for Moderating Pluralistic Communities on Social Media

arXiv:2605.17187v1 Announce Type: cross Abstract: Social media are shifting towards pluralism -- community-governed platforms where groups define their own norms. What violates rules in one community

SkillGenBench: Benchmarking Skill Generation Pipelines for LLM Agents

arXiv:2605.18693v1 Announce Type: new Abstract: As LLM agents are increasingly built around reusable skills, a central challenge is no longer only whether agents can use provided skills, but whether t

StyleText: A Large-Scale Dataset and Benchmark for Stylized Scene Text Inpainting

arXiv:2605.17309v1 Announce Type: cross Abstract: We present StyleText, a large-scale dataset and benchmark for localized scene-text inpainting with style preservation. StyleText contains 28,518 image

ToolMATH: A Diagnostic Benchmark for Long-Horizon Tool Use under Systematic Tool-Catalog Constraints

arXiv:2602.21265v2 Announce Type: replace Abstract: We introduce ToolMATH, a math-grounded diagnostic benchmark for evaluating long-horizon tool use under controllable tool-catalog conditions. ToolMAT