Search: “benchmark”

PlanSightRAG: A Visual-First Multimodal RAG for Automating Question Answering and Compliance Checking for Civil Standard Plans

arXiv:2608.26091v1 Announce Type: cross Abstract: Civil infrastructure compliance checking has long relied on engineers manually reading legacy 2D plans; however, OCR-based automation strips away the

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Reconstructing the Right Episode: Evaluating Interleaved Conversational Memory Beyond Long Context

DGX agent

arXiv:2608.25655v1 Announce Type: new Abstract: Conversations with chat assistants increasingly span many topics in a single long-running thread, challenging memory systems. Existing long-context and

model-releasesarxiv-cs-cl

27 Aug 2026

SciMIF: Understanding Multimodal Instruction Following in Scientific Domains

model-releasesarxiv-cs-lg

arXiv:2608.25973v1 Announce Type: cross Abstract: Understanding instruction-following capabilities in scientific domains is essential for effectively leveraging Multimodal Large Language Models (MLLMs

Unfolding Scientific Papers into Multi-Turn Generation Trajectories for Continued Pre-Training

arXiv:2608.25826v1 Announce Type: new Abstract: A recent line of synthetic-data work reconstructs the thinking behind existing text rather than rewriting the text itself, but it operates on short web

Zero-WAM: In-Context World-Action Modeling from Human Videos for Open-Ended Task Generalization

arXiv:2608.26103v1 Announce Type: cross Abstract: Zero-shot cross-task generalization, where a policy must execute manipulation tasks never seen during training, remains a central challenge in robot l

Comparing Uncertainty Measurement and Mitigation Methods for Large Language Models: A Systematic Review

arXiv:2504.18346v4 Announce Type: replace-cross Abstract: Large Language Models (LLMs) have been transformative across many domains. However, hallucination, i.e., confidently outputting incorrect info

Evaluating Multiple LLM Generations with Validated Task Coverage

arXiv:2608.24228v1 Announce Type: new Abstract: Many LLM applications are most useful when they provide several candidate outputs for comparison, validation, or combination. Predominant evaluation set

More GPUs or a Smaller Cache? Tensor Parallelism versus KV Compression for Memory-Bound LLM Serving

arXiv:2608.23962v1 Announce Type: new Abstract: When an LLM serving deployment runs out of KVcache room, there are two well-established ways out. Tensor parallelism shards the weights and the KV cache

TangramPuzzle: Evaluating Multimodal Large Language Models with Compositional Spatial Reasoning

arXiv:2601.16520v2 Announce Type: replace-cross Abstract: Multimodal Large Language Models (MLLMs) have achieved remarkable progress in visual recognition and semantic understanding, yet precise compo

ToolRobustBench: Stage-Wise Perturbation Evaluation and Failure Diagnosis for Tool-Calling Agents

arXiv:2608.23635v1 Announce Type: cross Abstract: Large language models (LLMs) rely on tool calling as a fundamental agent capability, enabling them to invoke external systems and complete tasks beyon

EviSafe: Evidence-Grounded Safety Evaluation for Vision-Language Models

arXiv:2608.23313v1 Announce Type: new Abstract: Vision-language model safety benchmarks typically evaluate only final responses: whether a model refuses, warns, or complies. This outcome-level view ca

25 Aug 2026

MedReaMM: Evaluating Large Multimodal Models on Expert-Level Clinical Diagnostic Synthesis

arXiv:2608.22323v1 Announce Type: new Abstract: The application of Large Language Models (LLMs) to diagnostic decision-making has garnered growing interest. However, existing benchmarks largely focus

25 Aug 2026

What's the Catch? Evaluating Temporal Consistency in Vision-Language Models

arXiv:2608.23474v1 Announce Type: cross Abstract: Vision-language models (VLMs) achieve strong performance on video and image-sequence benchmarks, yet it remains unclear whether they capture temporal

25 Aug 2026

Beyond Raw Transcripts: Structured Persona Extraction for LLM-Based Digital Twins

arXiv:2608.20344v1 Announce Type: new Abstract: LLM-based 'digital twins' aim to simulate how an individual would behavein new environments or respond to novel questions, given some representation of

Can Legal AI Know When It Is Wrong? And Do Students Know When It Is?

arXiv:2608.21089v1 Announce Type: new Abstract: Integrating Large Language Models (LLMs) into the Indian judiciary promises access to justice but introduces severe risks. We identify the 'inertia of c

Can Scientific Claims Be Removed from Large Language Models? A Systematic Evaluation of Claim-Level Unlearning

arXiv:2608.20960v1 Announce Type: new Abstract: Language models (LMs) are trained on static scientific corpora, whereas scientific knowledge continuously evolves through correction and revision. Scien

Do SpeechLMs Hear Their Own Opinions? Diagnosing and Mitigating Previous-Belief Contamination in Streaming Emotion Understanding

arXiv:2608.20769v1 Announce Type: cross Abstract: Streaming emotion understanding uses historical state while continuously interpreting current audio, often feeding the model's previous prediction bac

Free-Text Evaluation of LLMs for 5G Domain Knowledge and Fault Analysis using LLM-as-Judge

arXiv:2608.21021v1 Announce Type: cross Abstract: Real-world fault analysis in 5G and emerging 6G networks demands domain expertise to analyze free-text diagnostics, including root-cause explanations

Human-JEPA: A Human-Centric Vision Model that Perceives and Anticipates

arXiv:2608.21160v1 Announce Type: new Abstract: Machines that understand humans should perceive the present and anticipate the future. Existing human-centric vision model are pretrained on human image

KoViDoRe: Korean Visual Document Retrieval

arXiv:2608.20840v1 Announce Type: cross Abstract: Recent advances in multimodal retrieval have improved the ability to retrieve information from visually rich documents such as PDFs and reports. Howev

Natural-Language-Guided Generator-Agnostic Shortlisting for Protein Binder Design

arXiv:2608.20755v1 Announce Type: new Abstract: Modern de novo design workflows generate many candidate protein binders, but wet-lab validation capacity remains limited, making shortlisting a major bo

A Strong Linear Baseline for Whole-Heart Cardiac Shape Completion on CT, with an Open Eleven-Structure Statistical Shape Model

arXiv:2608.19932v1 Announce Type: new Abstract: Public cardiac cohorts annotate different subsets of the heart, so shapes from separate sources cannot be pooled without shared correspondence. Among re

BeyondMasks: Evaluating Causal and Physical Consistency in Video Object Removal

arXiv:2608.20107v1 Announce Type: new Abstract: Recent advances in generative video models have significantly improved visual realism in video object removal, yet evaluation protocols still focus on m

Can Agent Memory Systems Track Evolving State?

arXiv:2608.19652v1 Announce Type: new Abstract: As LLM-based agents are deployed for longer and higher-stakes tasks, their memory systems continue to have crucial gaps. While existing memory benchmark

GRACE: Grounded Reasoning via Adapter Composition and Evidence-Aware Calibration for Educational Visual Question Answering

arXiv:2608.19355v1 Announce Type: cross Abstract: Educational visual question answering, or VQA, requires models to solve curriculum-oriented multiple-choice questions using both language and visual e

PersonalBench: Measuring the Authorship Gap in LLM Personalization

arXiv:2608.19746v1 Announce Type: new Abstract: Personalized text generation aims to make LLMs write in a specific individual's style, yet existing benchmarks measure task accuracy or preference align

Remember, Verify, or Ask? Cross-Family Evaluation of Memory Commitment in LLM Agents

arXiv:2608.19564v1 Announce Type: new Abstract: Persistent memory can personalize an LLM agent, but an incorrect durable update can silently distort future behavior. We study the memory-clarification

Rule-Compliant Visual Spatial Planning for Multimodal Large Language Models

arXiv:2608.20237v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) combine linguistic reasoning with visual perception, yet their ability to perform visual spatial planning under

STEP: Score-Based Temporal Energy for Human Pose Video Anomaly Detection

arXiv:2608.19987v1 Announce Type: new Abstract: Skeleton-based Video Anomaly Detection (VAD) offers a robust, privacy-preserving solution for identifying abnormal behaviors. To model the distribution

Towards Clinically Faithful Medical Image Captioning via Enhanced Vision-Language Alignment

arXiv:2608.19825v1 Announce Type: cross Abstract: Medical image captioning is a technique that accelerates early-stage diagnostic workflows and enhances the interpretability of medical diagnostic AI s

A FEM-Based Surrogate Modelling and Optimization Framework for Physics-Constrained Electromagnetic Coil Design

model-releasesarxiv-cs-lg

arXiv:2608.18903v1 Announce Type: new Abstract: This work evaluates surrogate-assisted optimization of a seven-parameter current-excited coil--core benchmark subject to geometric, manufacturing, and s

FinSkillBench: Evaluating AI Agents and Domain Skills for Investment Management

arXiv:2608.18099v1 Announce Type: new Abstract: Investment management is a high-stakes domain in which agentic AI systems must do more than generate plausible text. They must retrieve point-in-time da

ForeSightGuide: An Anticipatory Framework toward Accurate and Low-Redundancy Guidance for the Visually Impaired

arXiv:2608.18993v1 Announce Type: new Abstract: Electronic travel aids are pivotal for the independent mobility of the visually impaired. While Vision-Language Models (VLMs) offer rich environmental u

MAVEN: A Macro-Societal Value Evaluation Framework of Multimodal Content with Compact Aligned Evaluators

arXiv:2608.18096v1 Announce Type: new Abstract: Assessing whether multimodal content aligns with macro-societal values, such as peace, justice, and freedom, has become an increasingly urgent challenge

MemFuse: Multi-Source Memory Fusion from Fragmented Observations

arXiv:2608.18704v1 Announce Type: cross Abstract: Long-term memory is essential for agents that operate across extended interactions, yet existing memory systems and benchmarks predominantly focus on

On the Robustness of Vision-Language Models in Zero-shot Privacy Classification

arXiv:2510.09253v2 Announce Type: replace Abstract: Automatic systems for document understanding require multimodal models that accurately identify sensitive visual content, even in the presence of im

StateTrace: An Object-Centric Framework for Hidden-State Spatiotemporal Reasoning in Long Videos

arXiv:2608.18532v1 Announce Type: new Abstract: Existing VLMs have achieved strong performance in video understanding, yet they struggle with long-video spatiotemporal reasoning when target objects be

Task-Conditioned Least-Privilege Learning for Executable Terminal and MCP Agents

arXiv:2608.18351v1 Announce Type: cross Abstract: Tool-using large language-model agents can complete a task while exercising authority that the user did not grant or the task does not need, causing e

UMER: Unifying Embedding and Ranking via Pair-Aware Discriminative Reasoning for Universal Multimodal Retrieval

arXiv:2608.18504v1 Announce Type: new Abstract: Universal multimodal retrieval aims to support diverse instruction-aware retrieval tasks, demanding both efficient corpus-scale matching and fine-graine

VTONQA: A Multi-Dimensional Quality Assessment Dataset for Virtual Try-on

arXiv:2601.02945v2 Announce Type: replace Abstract: With the rapid development of e-commerce and digital fashion, image-based virtual try-on (VTON) has attracted increasing attention. However, existin

Domain-Adapted Molecular Language Models for Efficient Search of Make-on-Demand Libraries

arXiv:2608.17567v1 Announce Type: cross Abstract: Pretrained molecular language models are increasingly used as molecular encoders for learning structure-property relationships. However, their practic

Effects of Answer Format Variation on Gender Bias in Large Language Models

arXiv:2608.17516v1 Announce Type: new Abstract: Gender bias or other social biases in large language models (LLMs) are frequently evaluated with question answering or survey benchmarks where the LLM n

Efficient Dynamic Shielding for Parametric Safety Specifications

arXiv:2505.22104v2 Announce Type: replace Abstract: Shielding has emerged as a promising approach for ensuring safety of AI-controlled autonomous systems. The algorithmic goal is to compute a shield,

Explicit State Elicitation Is Not Enough: A Controlled Audit of Memory-Policy Classification

arXiv:2608.17247v1 Announce Type: new Abstract: Personalized agents must decide whether retrieved user memory should be used, ignored, updated, or queried before it affects a current task. We use this

Foundation Agents Meet Agentic Deep Research: Evidence-Grounded Clinical Code Forecasting

arXiv:2608.17075v1 Announce Type: cross Abstract: Next-encounter ICD forecasting predicts which standardized diagnosis codes will be documented at a future visit from the longitudinal record available

PathoArgus: Advancing Evidence-Grounded Long-Context Visual Reasoning across Gigapixel Whole-Slide and Multi-Slide Case Contexts

arXiv:2608.17607v1 Announce Type: new Abstract: Whole-slide pathology reasoning requires models to integrate gigapixel-scale visual evidence across complete case-linked slides, yet current question-an

SW-ProxyCE: Zero-Query Adversarial Transfer from Public EEG Encoders to Private Downstream Models

model-releasesarxiv-cs-lg

arXiv:2608.16931v1 Announce Type: new Abstract: Electroencephalography (EEG) foundation models have recently emerged as a promising paradigm for EEG decoding by learning reusable representations from

Temporal Leakage in Financial News NLP: A Multi-Architecture Audit with a Regime-Specific M&A Signal

arXiv:2608.17223v1 Announce Type: new Abstract: Financial-news direction prediction has become a popular NLP benchmark, yet reported gains depend critically on whether the train-test split is chronolo