Search: “evaluation”

BhashaSutra: A Task-Centric Unified Survey of Indian NLP Datasets, Corpora, and Resources

arXiv:2604.18423v1 Announce Type: new Abstract: India's linguistic landscape, spanning 22 scheduled languages and hundreds of marginalized dialects, has driven rapid growth in NLP datasets, benchmarks

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Safety

BIASEDTALES-ML: A Multilingual Dataset for Analyzing Narrative Attribute Distributions in LLM-Generated Stories

DGX agent

arXiv:2604.17008v1 Announce Type: new Abstract: Large Language Models (LLMs) are increasingly used to generate narrative content, including children's stories, which play an important role in social a

safetyarxiv-cs-cl

21 Apr 2026

E3VS-Bench: A Benchmark for Viewpoint-Dependent Active Perception in 3D Gaussian Splatting Scenes

model-releasesarxiv-cs-cv

arXiv:2604.17969v1 Announce Type: new Abstract: Visual search in 3D environments requires embodied agents to actively explore their surroundings and acquire task-relevant evidence. However, existing v

How Robustly do LLMs Understand Execution Semantics?

model-releasesarxiv-cs-lg

arXiv:2604.16320v1 Announce Type: cross Abstract: LLMs demonstrate remarkable reasoning capabilities, yet whether they utilize internal world models or rely on sophisticated pattern matching remains o

IceBreaker for Conversational Agents: Breaking the First-Message Barrier with Personalized Starters

arXiv:2604.18375v1 Announce Type: new Abstract: Conversational agents, such as ChatGPT and Doubao, have become essential daily assistants for billions of users. To further enhance engagement, these sy

LiveFact: A Dynamic, Time-Aware Benchmark for LLM-Driven Fake News Detection

arXiv:2604.04815v2 Announce Type: replace Abstract: The rapid development of Large Language Models (LLMs) has transformed fake news detection and fact-checking tasks from simple classification to comp

MedProbeBench: Systematic Benchmarking at Deep Evidence Integration for Expert-level Medical Guideline

model-releasesarxiv-cs-cv

arXiv:2604.18418v1 Announce Type: new Abstract: Recent advances in deep research systems enable large language models to retrieve, synthesize, and reason over large-scale external knowledge. In medici

Model in Distress: Sentiment Analysis on French Synthetic Social Media

arXiv:2604.18226v1 Announce Type: new Abstract: Automated analysis of customer feedback on social media is hindered by three challenges: the high cost of annotated training data, the scarcity of evalu

PRISM: Probing Reasoning, Instruction, and Source Memory in LLM Hallucinations

arXiv:2604.16909v1 Announce Type: new Abstract: As large language models (LLMs) evolve from conversational assistants into agents capable of handling complex tasks, they are increasingly deployed in h

QU-NLP at QIAS 2026: Multi-Stage QLoRA Fine-Tuning for Arabic Islamic Inheritance Reasoning

arXiv:2604.16396v1 Announce Type: new Abstract: Islamic inheritance law (ilm al-mawar{i}th) presents a challenging domain for evaluating large language models' structured reasoning capabilities, requi

Reference-state System Reliability method for scalable uncertainty quantification of coherent systems

arXiv:2604.17066v1 Announce Type: new Abstract: Coherent systems are representative of many practical applications, ranging from infrastructure networks to supply chains. Probabilistic evaluation of s

researcharxiv-cs-lg

Revisiting a Pain in the Neck: A Semantic Reasoning Benchmark for Language Models

arXiv:2604.16593v1 Announce Type: new Abstract: We present SemanticQA, an evaluation suite designed to assess language models (LMs) in semantic phrase processing tasks. The benchmark consolidates exis

SafeVLA: Towards Safety Alignment of Vision-Language-Action Model via Constrained Learning

model-releasesarxiv-cs-ro

arXiv:2503.03480v4 Announce Type: replace Abstract: Vision-language-action models (VLAs) show potential as generalist robot policies. However, these models pose extreme safety challenges during real-w

TextTIGER: Text-based Intelligent Generation with Entity Prompt Refinement for Text-to-Image Generation

arXiv:2504.18269v2 Announce Type: replace Abstract: When generating images from prompts that include specific entities, the model must retain as much entity-specific knowledge as possible. However, th

When Choices Become Risks: Safety Failures of Large Language Models under Multiple-Choice Constraints

arXiv:2604.16916v1 Announce Type: new Abstract: Safety alignment in large language models (LLMs) is primarily evaluated under open-ended generation, where models can mitigate risk by refusing to respo

When Helpers Become Hazards: A Benchmark for Analyzing Multimodal LLM-Powered Safety in Daily Life

arXiv:2601.04043v2 Announce Type: replace Abstract: As Multimodal Large Language Models (MLLMs) become an indispensable assistant in human life, the unsafe content generated by MLLMs poses a danger to

When Misinformation Speaks and Converses: Rethinking Fact-Checking in Audio Platforms

arXiv:2604.16767v1 Announce Type: new Abstract: Audio platforms have evolved beyond entertainment. They have become central to public discourse, from podcasts and radio to WhatsApp voice notes and liv

Tutorials

A methodology to rank importance of frequencies and channels in electromyography data with Decision Tree classifiers

arXiv:2604.15353v1 Announce Type: cross Abstract: This study presents a methodology for identifying the most informative frequencies and channels in electromyography (EMG) data to evaluate muscle reco

tutorialsarxiv-cs-lg

CIG: Measuring Conversational Information Gain in Deliberative Dialogues with Semantic Memory Dynamics

arXiv:2604.15647v1 Announce Type: new Abstract: Measuring the quality of public deliberation requires evaluating not only civility or argument structure, but also the informational progress of a conve

DiZiNER: Disagreement-guided Instruction Refinement via Pilot Annotation Simulation for Zero-shot Named Entity Recognition

arXiv:2604.15866v1 Announce Type: cross Abstract: Large language models (LLMs) have advanced information extraction (IE) by enabling zero-shot and few-shot named entity recognition (NER), yet their ge

GTA-2: Benchmarking General Tool Agents from Atomic Tool-Use to Open-Ended Workflows

arXiv:2604.15715v1 Announce Type: cross Abstract: The development of general-purpose agents requires a shift from executing simple instructions to completing complex, real-world productivity workflows

Stargazer: A Scalable Model-Fitting Benchmark Environment for AI Agents under Astrophysical Constraints

model-releasesarxiv-cs-lg

arXiv:2604.15664v1 Announce Type: new Abstract: The rise of autonomous AI agents suggests that dynamic benchmark environments with built-in feedback on scientifically grounded tasks are needed to eval

Tabular foundation models for in-context prediction of molecular properties

arXiv:2604.16123v1 Announce Type: new Abstract: Accurate molecular property prediction is central to drug discovery, catalysis, and process design, yet real-world applications are often limited by sma

applicationsarxiv-cs-lg

VEFX-Bench: A Holistic Benchmark for Generic Video Editing and Visual Effects

arXiv:2604.16272v1 Announce Type: cross Abstract: As AI-assisted video creation becomes increasingly practical, instruction-guided video editing has become essential for refining generated or captured

An Underexplored Frontier: Large Language Models for Rare Disease Patient Education and Communication -- A scoping review

arXiv:2604.14179v1 Announce Type: new Abstract: Rare diseases affect over 300 million people worldwide and are characterized by complex care pathways, limited clinical expertise, and substantial unmet

Local Ai

Atropos: Improving Cost-Benefit Trade-off of LLM-based Agents under Self-Consistency with Early Termination and Model Hotswap

arXiv:2604.15075v1 Announce Type: cross Abstract: Open-weight Small Language Models(SLMs) can provide faster local inference at lower financial cost, but may not achieve the same performance level as

local-aiarxiv-cs-lg

De-Anonymization at Scale via Tournament-Style Attribution

arXiv:2601.12407v2 Announce Type: replace-cross Abstract: As LLMs rapidly advance and enter real-world use, their privacy implications are increasingly important. We study an authorship de-anonymizati

Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations

arXiv:2604.15302v1 Announce Type: cross Abstract: LLM-as-judge frameworks are increasingly used for automatic NLG evaluation, yet their per-instance reliability remains poorly understood. We present a

Internal Knowledge Without External Expression: Probing the Generalization Boundary of a Classical Chinese Language Model

arXiv:2604.14180v1 Announce Type: new Abstract: We train a 318M-parameter Transformer language model from scratch on a curated corpus of 1.56 billion tokens of pure Classical Chinese, with zero Englis

Learning to Think Like a Cartoon Captionist: Incongruity-Resolution Supervision for Multimodal Humor Understanding

arXiv:2604.15210v1 Announce Type: cross Abstract: Humor is one of the few cognitive tasks where getting the reasoning right matters as much as getting the answer right. While recent work evaluates hum

Meituan Merchant Business Diagnosis via Policy-Guided Dual-Process User Simulation

arXiv:2604.15190v1 Announce Type: cross Abstract: Simulating group-level user behavior enables scalable counterfactual evaluation of merchant strategies without costly online experiments. However, bui

Mitigating LLM biases toward spurious social contexts using direct preference optimization

arXiv:2604.02585v2 Announce Type: replace-cross Abstract: LLMs are increasingly used for high-stakes decision-making, yet their sensitivity to spurious contextual information can introduce harmful bia

Standard-to-Dialect Transfer Trends Differ across Text and Speech: A Case Study on Intent and Topic Classification in German Dialects

arXiv:2510.07890v3 Announce Type: replace Abstract: Research on cross-dialectal transfer from a standard to a non-standard dialect variety has typically focused on text data. However, dialects are pri

TwinOR: Photorealistic Digital Twins of Dynamic Operating Rooms for Embodied AI Research

arXiv:2511.07412v2 Announce Type: replace Abstract: Developing embodied AI for intelligent surgical systems requires safe, controllable environments for continual learning and evaluation. However, saf

safetyarxiv-cs-cv

VoxSafeBench: Not Just What Is Said, but Who, How, and Where

arXiv:2604.14548v1 Announce Type: cross Abstract: As speech language models (SLMs) transition from personal devices into shared, multi-user environments, their responses must account for far more than

safetyarxiv-cs-lg

EmbodiedClaw: Conversational Workflow Execution for Embodied AI Development

model-releasesarxiv-cs-ro

arXiv:2604.13800v1 Announce Type: new Abstract: Embodied AI research is increasingly moving beyond single-task, single-environment policy learning toward multi-task, multi-scene, and multi-model setti

Empirical Evidence of Complexity-Induced Limits in Large Language Models on Finite Discrete State-Space Problems with Explicit Validity Constraints

arXiv:2604.13371v1 Announce Type: new Abstract: Large Language Models (LLMs) are increasingly described as possessing strong reasoning capabilities, supported by high performance on mathematical, logi

Giving Voice to the Constitution: Low-Resource Text-to-Speech for Quechua and Spanish Using a Bilingual Legal Corpus

arXiv:2604.13288v1 Announce Type: new Abstract: We present a unified pipeline for synthesizing high-quality Quechua and Spanish speech for the Peruvian Constitution using three state-of-the-art text-t

HINTBench: Horizon-agent Intrinsic Non-attack Trajectory Benchmark

model-releasesarxiv-cs-lg

arXiv:2604.13954v1 Announce Type: new Abstract: Existing agent-safety evaluation has focused mainly on externally induced risks. Yet agents may still enter unsafe trajectories under benign conditions.

Scaling Test-Time Compute to Achieve IOI Gold Medal with Open-Weight Models

arXiv:2510.14232v2 Announce Type: replace-cross Abstract: Competitive programming has become a rigorous benchmark for evaluating the reasoning and problem-solving capabilities of large language models

BEAM: Bi-level Memory-adaptive Algorithmic Evolution for LLM-Powered Heuristic Design

arXiv:2604.12898v1 Announce Type: new Abstract: Large Language Model-based Hyper Heuristic (LHH) has recently emerged as an efficient way for automatic heuristic design. However, most existing LHHs ju

researcharxiv-cs-ai

GRADE: Probing Knowledge Gaps in LLMs through Gradient Subspace Dynamics

arXiv:2604.02830v2 Announce Type: replace Abstract: Detecting whether a model's internal knowledge is sufficient to correctly answer a given question is a fundamental challenge in deploying responsibl

Human-Inspired Context-Selective Multimodal Memory for Social Robots

arXiv:2604.12081v1 Announce Type: new Abstract: Memory is fundamental to social interaction, enabling humans to recall meaningful past experiences and adapt their behavior accordingly based on the con

researcharxiv-cs-ai

Red Teaming Large Reasoning Models

arXiv:2512.00412v4 Announce Type: replace-cross Abstract: Large Reasoning Models (LRMs) have emerged as a powerful advancement in multi-step reasoning tasks, offering enhanced transparency and logical

A Benchmark for Gap and Overlap Analysis as a Test of KG Task Readiness

arXiv:2604.10853v1 Announce Type: new Abstract: Task-oriented evaluation of knowledge graph (KG) quality increasingly asks whether an ontology-based representation can answer the competency questions

A Dual-Positive Monotone Parameterization for Multi-Segment Bids and a Validity Assessment Framework for Reinforcement Learning Agent-based Simulation of Electricity Markets

arXiv:2604.10252v1 Announce Type: new Abstract: Reinforcement learning agent-based simulation (RL-ABS) has become an important tool for electricity market mechanism analysis and evaluation. In the mod

safetyarxiv-cs-ai

ACCIDENT: A Benchmark Dataset for Vehicle Accident Detection from Traffic Surveillance Videos

arXiv:2604.09819v1 Announce Type: cross Abstract: We introduce ACCIDENT, a benchmark dataset for traffic accident detection in CCTV footage, designed to evaluate models in supervised (IID and OOD) and

AgencyBench: Benchmarking the Frontiers of Autonomous Agents in 1M-Token Real-World Contexts

arXiv:2601.11044v3 Announce Type: replace Abstract: Large Language Models (LLMs) based autonomous agents demonstrate multifaceted capabilities to contribute substantially to economic production. Howev