Search: “evaluation”

Alexandria: A Multi-Domain Dialectal Arabic Machine Translation Dataset for Culturally Inclusive and Linguistically Diverse LLMs

arXiv:2601.13099v2 Announce Type: replace Abstract: Arabic is a highly diglossic language where most daily communication occurs in regional dialects rather than Modern Standard Arabic (MSA). Despite t

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Tutorials

Argument Reconstruction as Supervision for Critical Thinking in LLMs

DGX agent

arXiv:2603.17432v2 Announce Type: replace Abstract: To think critically about arguments, human learners are trained to identify, reconstruct, and evaluate arguments. Argument reconstruction is especia

tutorialsarxiv-cs-cl

21 Apr 2026

BASIL: Bayesian Assessment of Sycophancy in LLMs

arXiv:2508.16846v5 Announce Type: replace-cross Abstract: Sycophancy (overly agreeable or flattering behavior) poses a fundamental challenge for human-AI collaboration, particularly in high-stakes dec

applicationsarxiv-cs-cl

Benchmarking System Dynamics AI Assistants: Cloud Versus Local LLMs on CLD Extraction and Discussion

arXiv:2604.18566v1 Announce Type: cross Abstract: We present a systematic evaluation of large language model families -- spanning both proprietary cloud APIs and locally-hosted open-source models -- o

Beyond Reproduction: A Paired-Task Framework for Assessing LLM Comprehension and Creativity in Literary Translation

arXiv:2604.18169v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used for creative tasks such as literary translation. Yet translational creativity remains underexplored a

BridgeEQA: Virtual Embodied Agents for Real Bridge Inspections

arXiv:2511.12676v2 Announce Type: replace Abstract: Deploying embodied agents that can answer questions about their surroundings in realistic real-world settings remains difficult, partly due to the s

Calibrated? Not for Everyone: How Sexual Orientation and Religious Markers Distort LLM Accuracy and Confidence in Medical QA

arXiv:2604.17316v1 Announce Type: new Abstract: Safe clinical deployment of Large Language Models (LLMs) requires not only high accuracy but also robust uncertainty calibration to ensure models defer

applicationsarxiv-cs-cl

Concurrent Criterion Validation of a Validity Screen for LLM Confidence Signals via Selective Prediction

arXiv:2604.17716v1 Announce Type: new Abstract: The validity screen (Cacioli, 2026d, 2026e) classifies LLM confidence signals as Valid, Indeterminate, or Invalid. We test whether these classifications

Culture-Aware Humorous Captioning: Multimodal Humor Generation across Cultural Contexts

arXiv:2604.18091v1 Announce Type: new Abstract: Recent multimodal large language models have shown promising ability in generating humorous captions for images, yet they still lack stable control over

DSH-Bench: A Difficulty- and Scenario-Aware Benchmark with Hierarchical Subject Taxonomy for Subject-Driven Text-to-Image Generation

arXiv:2603.08090v2 Announce Type: replace Abstract: Significant progress has been achieved in subject-driven text-to-image (T2I) generation, which aims to synthesize new images depicting target subjec

Fairness Constraints in High-Dimensional Generalized Linear Models

arXiv:2604.16610v1 Announce Type: cross Abstract: Machine learning models often inherit biases from historical data, raising critical concerns about fairness and accountability. Conventional fairness

safetyarxiv-cs-lg

'Faithful to What?' On the Limits of Fidelity-Based Explanations

arXiv:2506.12176v5 Announce Type: replace Abstract: In explainable AI, surrogate models are commonly evaluated by their fidelity to a neural network's predictions. Fidelity, however, measures alignmen

safetyarxiv-cs-lg

Hard to Be Heard: Phoneme-Level ASR Analysis of Phonologically Complex, Low-Resource Endangered Languages

arXiv:2604.18204v1 Announce Type: new Abstract: We present a phoneme-level analysis of automatic speech recognition (ASR) for two low-resourced and phonologically complex East Caucasian languages, Arc

HiRAS: A Hierarchical Multi-Agent Framework for Paper-to-Code Generation and Execution

arXiv:2604.17745v1 Announce Type: new Abstract: Recent advances in large language models have highlighted their potential to automate computational research, particularly reproducing experimental resu

HORIZON: A Benchmark for In-the-wild User Behaviour Modeling

arXiv:2604.17259v1 Announce Type: cross Abstract: User behavior in the real world is diverse, cross-domain, and spans long time horizons. Existing user modeling benchmarks however remain narrow, focus

Human-Centered Supervision for Sentiment Analysis in Telugu: A Systematic Inquiry Beyond Accuracy

arXiv:2508.01486v3 Announce Type: replace Abstract: Sentiment analysis for low-resource languages remains challenging in an era where interpretability, human alignment, and fairness are increasingly n

IDOBE: Infectious Disease Outbreak forecasting Benchmark Ecosystem

arXiv:2604.18521v1 Announce Type: new Abstract: Epidemic forecasting has become an integral part of real-time infectious disease outbreak response. While collaborative ensembles composed of statistica

Is SAM3 ready for pathology segmentation?

arXiv:2604.18225v1 Announce Type: new Abstract: Is Segment Anything Model 3 (SAM3) capable in segmenting Any Pathology Images? Digital pathology segmentation spans tissue-level and nuclei-level scales

researcharxiv-cs-cv

Live Avatar: Streaming Real-time Audio-Driven Avatar Generation with Infinite Length

arXiv:2512.04677v5 Announce Type: replace Abstract: Audio-driven avatar interaction demands real-time, streaming, and infinite-length generation -- capabilities fundamentally at odds with the sequenti

MedPRMBench: A Fine-grained Benchmark for Process Reward Models in Medical Reasoning

arXiv:2604.17282v1 Announce Type: new Abstract: Process-Level Reward Models (PRMs) are essential for guiding complex reasoning in large language models, yet existing PRM benchmarks cover only general

MetaLint: Easy-to-Hard Generalization for Code Linting

arXiv:2507.11687v4 Announce Type: replace-cross Abstract: Large language models excel at code generation but struggle with code linting, particularly in generalizing to unseen or evolving best practic

Mix and Match: Context Pairing for Scalable Topic-Controlled Educational Summarisation

arXiv:2604.18087v1 Announce Type: new Abstract: Topic-controlled summarisation enables users to generate summaries focused on specific aspects of source documents. This paper investigates a data augme

Agents

MODEST: Multi-Optics Depth-of-Field Stereo Dataset

arXiv:2511.20853v3 Announce Type: replace Abstract: Reliable depth estimation under real optical conditions remains a core challenge for camera vision in systems such as autonomous robotics and augmen

agentsarxiv-cs-cv

No One Fits All: From Fixed Prompting to Learned Routing in Multilingual LLMs

arXiv:2604.16937v1 Announce Type: new Abstract: Translation-based prompting is widely used in multilingual LLMs, yet its effectiveness varies across languages and tasks. We evaluate prompting strategi

P-Check: Advancing Personalized Reward Model via Learning to Generate Dynamic Checklist

arXiv:2601.02986v2 Announce Type: replace Abstract: Recent approaches in personalized reward modeling have primarily focused on leveraging user interaction history to align model judgments with indivi

PFDelta: A Benchmark Dataset for Power Flow under Load, Generation, and Topology Variations

arXiv:2510.22048v3 Announce Type: replace Abstract: Power flow (PF) calculations are the backbone of real-time grid operations, across workflows such as contingency analysis (where repeated PF evaluat

Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?

arXiv:2604.17338v1 Announce Type: cross Abstract: Unlike code completion, debugging requires localizing faults and applying targeted edits. We observe that frontier LLMs often regenerate correct but o

Predicting LLM Compression Degradation from Spectral Statistics

arXiv:2604.18085v1 Announce Type: new Abstract: Matrix-level low-rank compression is a promising way to reduce the cost of large language models, but running compression and evaluating the resulting m

Pref-GRPO: Pairwise Preference Reward-based GRPO for Stable Text-to-Image Reinforcement Learning

arXiv:2508.20751v2 Announce Type: replace Abstract: Recent advancements highlight the importance of GRPO-based reinforcement learning methods and benchmarking in enhancing text-to-image (T2I) generati

Process Reward Models Meet Planning: Generating Precise and Scalable Datasets for Step-Level Rewards

arXiv:2604.17957v1 Announce Type: new Abstract: Process Reward Models (PRMs) have emerged as a powerful tool for providing step-level feedback when evaluating the reasoning of Large Language Models (L

Progressive Online Video Understanding with Evidence-Aligned Timing and Transparent Decisions

arXiv:2604.18459v1 Announce Type: new Abstract: Visual agents operating in the wild must respond to queries precisely when sufficient evidence first appears in a video stream, a critical capability th

SpeechMedAssist: Efficiently and Effectively Adapting Speech Language Models for Medical Consultation

arXiv:2601.04638v2 Announce Type: replace Abstract: Medical consultations are intrinsically speech-centric. However, most prior works focus on long-text-based interactions, which are cumbersome and pa

SPENCE: A Syntactic Probe for Detecting Contamination in NL2SQL Benchmarks

arXiv:2604.17771v1 Announce Type: new Abstract: Large language models (LLMs) have achieved strong performance on natural language to SQL (NL2SQL) benchmarks, yet their reported accuracy may be inflate

Synthia: Scalable Grounded Persona Generation from Social Media Data

arXiv:2507.14922v2 Announce Type: replace Abstract: Persona-driven simulations are increasingly used in computational social science, yet their validity critically depends on the fidelity of the under

Systematic Capability Benchmarking of Frontier Large Language Models for Offensive Cyber Tasks

arXiv:2604.17159v1 Announce Type: cross Abstract: We present, to our knowledge, the most comprehensive cross-model evaluation of LLM agents on offensive cybersecurity tasks, benchmarking 10 frontier m

TeleEmbedBench: A Multi-Corpus Embedding Benchmark for RAG in Telecommunications

arXiv:2604.17778v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed in the telecommunications domain for critical tasks, relying heavily on Retrieval-Augmented Gener

Thermal-GEMs: Generalized Models for Building Thermal Dynamics

arXiv:2604.16443v1 Announce Type: cross Abstract: Data-driven models for building thermal dynamics are a scalable approach for enabling energy-efficient operation through fault detection & diagnosis o

applicationsarxiv-cs-lg

Training-inference input alignment outweighs framework choice in longitudinal retinal image prediction

arXiv:2604.16955v1 Announce Type: new Abstract: Quantitative prediction of future retinal appearance from longitudinal imaging would support clinical decisions in progressive macular disease that curr

safetyarxiv-cs-cv

Trustworthy deep domain adaptation for wearable photoplethysmography signal analysis with decision-theoretic uncertainty quantification

arXiv:2604.17480v1 Announce Type: new Abstract: In principle, deep generative models can be used to perform domain adaptation; i.e. align the input feature representations of test data with that of a

applicationsarxiv-cs-lg

Using large language models for embodied planning introduces systematic safety risks

arXiv:2604.18463v1 Announce Type: cross Abstract: Large language models are increasingly used as planners for robotic systems, yet how safely they plan remains an open question. To evaluate safe plann

WeatherArchive-Bench: Benchmarking Retrieval-Augmented Reasoning for Historical Weather Archives

arXiv:2510.05336v2 Announce Type: replace Abstract: Historical archives on weather events are collections of enduring primary source records that offer rich, untapped narratives of how societies have

ASMR-Bench: Auditing for Sabotage in ML Research

model-releasesarxiv-cs-ai

arXiv:2604.16286v1 Announce Type: new Abstract: As AI systems are increasingly used to conduct research autonomously, misaligned systems could introduce subtle flaws that produce misleading results wh

AutoNFS: Automatic Neural Feature Selection

arXiv:2503.13304v2 Announce Type: replace Abstract: Feature selection (FS) is a fundamental challenge in machine learning, particularly for high-dimensional tabular data, where interpretability and co

applicationsarxiv-cs-lg

BAGEL: Benchmarking Animal Knowledge Expertise in Language Models

model-releasesarxiv-cs-ai

arXiv:2604.16241v1 Announce Type: cross Abstract: Large language models have shown strong performance on broad-domain knowledge and reasoning benchmarks, but it remains unclear how well language model

ConlangCrafter: Constructing Languages with a Multi-Hop LLM Pipeline

arXiv:2508.06094v4 Announce Type: replace Abstract: Constructed languages (conlangs) such as Esperanto and Quenya have played diverse roles in art, philosophy, and international communication. Meanwhi

Cut Your Losses! Learning to Prune Paths Early for Efficient Parallel Reasoning

arXiv:2604.16029v1 Announce Type: new Abstract: Parallel reasoning enhances Large Reasoning Models (LRMs) but incurs prohibitive costs due to futile paths caused by early errors. To mitigate this, pat

applicationsarxiv-cs-cl

CXR-LT 2026 Challenge: Multi-Center Long-Tailed and Zero Shot Chest X-ray Classification

arXiv:2604.15555v1 Announce Type: new Abstract: Chest X-ray (CXR) interpretation is hindered by the long-tailed distribution of pathologies and the open-world nature of clinical environments. Existing

Agents

DeepER-Med: Advancing Deep Evidence-Based Research in Medicine Through Agentic AI

arXiv:2604.15456v1 Announce Type: new Abstract: Trustworthiness and transparency are essential for the clinical adoption of artificial intelligence (AI) in healthcare and biomedical research. Recent d

agentsarxiv-cs-ai