Search: “evaluation”

Compliance, Capability, and Conflict: Benchmarking Multimodal LLMs under System Messages

arXiv:2608.19207v1 Announce Type: new Abstract: Production deployments of Multimodal Large Language Models (MLLMs) increasingly rely on system messages to govern model behavior. Yet existing benchmark

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

ConceptGuard: Benchmarking Context-Sensitive Unlearning in Large Language Models

DGX agent

arXiv:2608.20338v1 Announce Type: new Abstract: Large Language Models (LLMs) increasingly require selective removal of harmful or sensitive knowledge, called unlearning, yet existing methods and bench

model-releasesarxiv-cs-cl

21 Aug 2026

DiverValue-Bench: A Benchmark and Fine-Tuning Framework for Aligning Large Language Models with Diverse Human Values

arXiv:2509.08022v3 Announce Type: replace-cross Abstract: Aligning large language models (LLMs) with diverse human values is essential for safe and effective deployment, yet existing benchmarks often

ExPhy: A Benchmark for Explicit Physical Property Learning in Multi-Object Trajectory Forecasting

arXiv:2608.20009v1 Announce Type: new Abstract: Understanding object dynamics requires not only predicting future trajectories but also examining whether a model captures the physical properties that

Explainable Multimodal Depression Recognition in Clinical Interviews via PHQ-Aligned Symptom Summarization

arXiv:2501.16106v2 Announce Type: replace Abstract: Recent advances in multimodal depression recognition for clinical interviews (MDRC) have demonstrated the potential of AI systems by integrating tex

researcharxiv-cs-cl

Learning the Right Abstraction: Neural Reduced Dynamics for Complex Robot Control

arXiv:2608.19375v1 Announce Type: new Abstract: High-fidelity embodied AI simulators provide realistic evaluation of complex robotic systems, but their computational cost limits their direct use for l

safetyarxiv-cs-ro

Mapping General-Purpose AI Governance in Twenty AI Middle-Power Jurisdictions

arXiv:2608.19278v1 Announce Type: cross Abstract: The most capable general-purpose AI (GPAI) models are mostly built in two jurisdictions, the United States and China, but the risks they carry land gl

Maximum Likelihood Reinforcement Learning

arXiv:2602.02710v2 Announce Type: replace Abstract: Reinforcement learning (RL) is the method of choice for training models in setups where the objective function can only be evaluated by sampling fro

researcharxiv-cs-lg

Tutorials

MemTrapBench: Benchmarking Cognitive Traps in LLM Memory Use

arXiv:2608.20202v1 Announce Type: new Abstract: Memory has become a key component of large language models, enabling them to retain information and learn from long-term interactions. However, existing

tutorialsarxiv-cs-ai

NepOOC-M: Bilingual Nepali-English Benchmark and Comparative Analysis of Multimodal Architectures for OOC Detection

arXiv:2608.19212v1 Announce Type: new Abstract: Out-of-context (OOC) misinformation pairs authentic images with misleading captions to construct false narratives without image manipulation, making det

Quantum Kernel Estimation for the Discovery of Early Lung Cancer Detection

arXiv:2608.19304v1 Announce Type: cross Abstract: Lung cancer screening with low-dose chest computed tomography reduces mortality, but its impact is limited by uptake, adherence, and management challe

Repo0: Design-Driven Zero-to-All Code Generation

arXiv:2608.19854v1 Announce Type: cross Abstract: Large language model agents have made substantial progress in code generation, yet most existing systems assume a predefined repository architecture.

SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?

arXiv:2608.19799v1 Announce Type: new Abstract: Software increasingly functions as part of the scientific instrument itself, making failures in scientific code capable of compromising not only program

TempJail: Temporal Jailbreak Attack against Large Vision-Language Models via Subtitle Scheduling

arXiv:2608.19737v1 Announce Type: cross Abstract: Large vision-language models (LVLMs) have achieved remarkable progress in video understanding and reasoning. Despite extensive studies on text- and im

Towards Quantifying Benchmark Optimization in ASR Models

arXiv:2608.19936v1 Announce Type: cross Abstract: Public benchmarks are important measures of Automatic Speech Recognition (ASR) model capabilities. However, by nature of being public, there is risk o

Towards Surgical World-Action Modeling: A Preliminary Joint Visual-Trajectory Forecasting for Surgical Motion Planning

arXiv:2608.20284v1 Announce Type: new Abstract: Reliable surgical planning requires models to anticipate not only how instruments will move, but also how the operative visual state will evolve togethe

researcharxiv-cs-cv

When Guidance Goes Off-Scale: Recalibrating Diffusion Transformers under Analog Compute-in-Memory Nonidealities

arXiv:2608.19644v1 Announce Type: new Abstract: Diffusion Transformers (DiTs) incur high memory traffic and energy costs because sampling repeatedly evaluates large denoisers dominated by linear opera

researcharxiv-cs-cv

When Saying No Makes Better Videos: Designing Dual Gatekeeping for Pedagogically Grounded AI Content Creation

arXiv:2608.19812v1 Announce Type: new Abstract: To prevent the adoption of aesthetically polished but pedagogically flawed AI content, we study a video authoring pipeline featuring two layers of struc

Which Eviction Policy Should an LLM Cache Use? A Systematic Study Across Workloads, Capacities, and Encoders

arXiv:2608.20280v1 Announce Type: cross Abstract: Semantic caches reuse an LLM response when the incoming query embedding lies near a cached query, but proposed eviction policies have rarely been comp

safetyarxiv-cs-lg

Agents

A Multi-Agent Platform for Automated Enterprise Analytics and Insight Generation

arXiv:2608.18740v1 Announce Type: new Abstract: This paper proposes a multi-agent framework built on CrewAI [1] for conversational business intelligence. Five specialized AI agents operate in a sequen

agentsarxiv-cs-ai

A systematic review of machine learning techniques to address diagnosis and treatment of autism: challenges and opportunities

arXiv:2608.18188v1 Announce Type: cross Abstract: Autism spectrum disorder (ASD) is a developmental disability characterized by challenges in social interaction and communication. As the causes of ASD

Aslema at NADI 2026: Augmentation through Fewshot for SLU

arXiv:2608.18689v1 Announce Type: cross Abstract: We present Aslema, our system for NADI 2026 Shared Task 5, which consists of two subtasks: intent recognition and slot filling. We evaluate four omni

DeepTCM1.0: A Multi-Expert AI Agent for Deciphering Mechanisms of Chinese Herbal Formulae Based on General Large Language Models

arXiv:2608.18103v1 Announce Type: cross Abstract: Background: Mechanistic elucidation of traditional Chinese medicine (TCM) compound formulas remains a central challenge in the modernization of TCM. C

FairGlucose: A CGM Fairness Benchmark Reveals Subgroup Disparities Hidden in Population-Level Validation

arXiv:2608.18296v1 Announce Type: cross Abstract: As CGM-based AI tools approach clinical deployment, whether their accuracy is equitable across patient demographics remains insufficiently tested. To

FM-Bench: A Benchmark for Long-Horizon Management with Competing Agents

arXiv:2608.18423v1 Announce Type: new Abstract: Language model agents now execute bounded tasks reliably. Whether they can sustain effective decision-making over long horizons, where actions have cumu

How Quantum Is the Advantage? A Fair, Calibration- and Noise-Aware Benchmark and Attribution Audit of Quantum Machine Learning for Network Intrusion Detection

arXiv:2608.18155v1 Announce Type: cross Abstract: Quantum machine learning (QML) for network intrusion detection (NIDS) is routinely reported to reach near-perfect accuracy, yet the most rigorous stud

MedUAG: Unified Understanding and Generation for Medical Multimodal Models

arXiv:2608.18937v1 Announce Type: cross Abstract: Recent Multimodal Large Language Models (MLLMs) are rapidly evolving into unified understanding and generation (UAG) frameworks. However, extending th

Applications

Optimizing Energy Efficiency and Grid Stability via Public EV Charging Flexibility

arXiv:2608.18126v1 Announce Type: cross Abstract: This study evaluates the potential of electric vehicle (EV) charging flexibility to enhance both energy efficiency and power grid stability. Using rea

applicationsarxiv-cs-lg

Persona-Guided LLM Agents for Task-Oriented Dialogue

arXiv:2608.18085v1 Announce Type: new Abstract: Prior work has shown that large language models (LLMs) can express diverse personality traits in open-ended text generation. However, it remains unclear

Position: Current Model Cards Are Insufficient for Downstream Governance of Open-Weight Foundation Models

arXiv:2608.18086v1 Announce Type: new Abstract: The growth of open-weight foundation models (OWFMs) has prompted the AI community to re-evaluate strategies for effective downstream governance. Althoug

safetyarxiv-cs-ai

Agents

Self-Improvement of Large Language Models: A Technical Overview and Future Outlook

arXiv:2603.25681v2 Announce Type: replace Abstract: As large language models (LLMs) continue to advance, improving them solely through human supervision is becoming increasingly costly and limited in

agentsarxiv-cs-cl

Solving Is Not Drawing: A Benchmark for Diagrammatic Reasoning in Olympiad Geometry

arXiv:2608.18111v1 Announce Type: new Abstract: Foundation models such as GPT and Claude now solve olympiad-level mathematics with remarkable proficiency, so much so that geometry problem solving has

Structure, Association, and Decision Value: Representation-Based Difficulty Estimation for Adaptive Inference in African-Language NLI

arXiv:2608.19003v1 Announce Type: new Abstract: We ask whether internal representation statistics can provide useful example-level difficulty signals for adaptive inference in multilingual African NLP

Tangut Word Segmentation under Extreme Resource Scarcity: Integrating Traditional Lexicons and Unlabeled Text

arXiv:2608.18437v1 Announce Type: new Abstract: Tangut is an extinct language whose script does not explicitly mark word boundaries. We present the first systematic study of Tangut word segmentation u

researcharxiv-cs-cl

The Impact of CutMix on Reliability and Robustness in Semantic Segmentation

arXiv:2608.18715v1 Announce Type: cross Abstract: Ensuring not only high accuracy but also reliable and robust predictions is critical for the deployment of semantic segmentation models in safety-crit

safetyarxiv-cs-ai

VA-Judger: Reward Modeling from Human Preference Feedback for Joint Video-Audio Generation

model-releasesarxiv-cs-cv

arXiv:2608.18607v1 Announce Type: new Abstract: Using reinforcement learning to post-train joint video-audio generation models requires a reward signal. Existing methods construct this reward by combi

VTONQA: A Multi-Dimensional Quality Assessment Dataset for Virtual Try-on

model-releasesarxiv-cs-cv

arXiv:2601.02945v2 Announce Type: replace Abstract: With the rapid development of e-commerce and digital fashion, image-based virtual try-on (VTON) has attracted increasing attention. However, existin

What is Missing from AI Post-Training AI: An Empirical Analysis

arXiv:2608.19072v1 Announce Type: new Abstract: Large language model (LLM) agents can now post-train an LLM end-to-end. They can write code, launch training, evaluate checkpoints, and improve downstre

A decodability criterion predicts when hidden-state selection beats majority voting in large language models

arXiv:2608.17124v1 Announce Type: new Abstract: Combining the answers a large language model (LLM) samples for a question into one decision is a test-time information fusion problem, usually solved by

Advancing Health Equity through Multi-Level Fairness in Health Informatics

arXiv:2608.16902v1 Announce Type: cross Abstract: The increasing integration of machine learning in healthcare has highlighted critical challenges related to fairness, transparency, and health equity.

safetyarxiv-cs-lg

An Empirical Study of Reward Specification and Benchmark Reliability in GRPO-based LLM Unlearning

arXiv:2608.17804v1 Announce Type: cross Abstract: Practical LLM unlearning is usually evaluated through two objectives: suppress target-specific knowledge and preserve non-target utility. In generativ

ASI-Bench: At the Dawn of Artificial Superintelligence

arXiv:2608.17271v1 Announce Type: new Abstract: Artificial superintelligence (ASI) requires AI to move beyond mastering existing knowledge toward exploring the unknown, creating new knowledge, and tur

BEAR-Bench: A Bilingual Enterprise and Academic Reasoning Benchmark for Multimodal Models

arXiv:2608.17895v1 Announce Type: cross Abstract: While Multimodal Large Language Models (MLLMs) have made significant strides in visual comprehension, their ability to reason about text-dense, profes

Can Large Language Models Explain Flight Safety Events? A Prior-Guided Semantic LLM-based Approach

arXiv:2608.18017v1 Announce Type: new Abstract: Improving flight safety with flight data requires not only accurate detection of risk events, but more importantly, clear interpretation of their underl

safetyarxiv-cs-ai

Digital Twin-Based Intrusion Detection for Vehicle Powertrain CAN Bus Systems

arXiv:2608.17093v1 Announce Type: cross Abstract: Existing automotive intrusion detection systems (IDSs) for the Controller Area Network (CAN) largely target discrepancies in message timing, frequency

researcharxiv-cs-lg

Does Unification Come at a Cost? Uni-SafeBench: A Safety Benchmark for Unified Multimodal Large Models

arXiv:2604.00547v2 Announce Type: replace Abstract: Unified Multimodal Large Models (UMLMs) integrate understanding and generation capabilities within a single architecture. While unified architecture

Exploring Efficient Open-Vocabulary Segmentation in the Remote Sensing

arXiv:2509.12040v3 Announce Type: replace-cross Abstract: Open-Vocabulary Remote Sensing Image Segmentation (OVRSIS), an emerging task that adapts Open-Vocabulary Segmentation (OVS) to the remote sens

Tutorials

From Corpora to Co-Evolving Capabilities: Capability-Centric Data Design for Generalist Image Generation

arXiv:2608.18076v1 Announce Type: cross Abstract: Large-scale image generation has benefited from advances in data scale, quality, rebalancing, and recaptioning, yet conventional pipelines typically o

tutorialsarxiv-cs-ai