Search: “evaluation”

Towards AI-Assisted Research Writing: Benchmarking LLMs for AI/ML Introduction Generation

arXiv:2508.14273v3 Announce Type: replace Abstract: As researchers increasingly adopt LLMs as writing assistants, generating high-quality research paper introductions remains both challenging and esse

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Towards Stream Learning on Embedded Systems: Benchmarking the Memory Consumption of Stream Learning Methods

DGX agent

arXiv:2608.30923v1 Announce Type: cross Abstract: Stream learning is commonly evaluated through predictive performance and adaptation to concept drift. However, sustained operation of a stream learner

model-releasesarxiv-cs-ai

1 Sept 2026

UReason: Benchmarking Reasoning-to-Generation Alignment in Unified Multimodal Models

arXiv:2602.08336v3 Announce Type: replace Abstract: Unified multimodal models (UMMs) aim to integrate multimodal understanding and generation within a unified architecture, yet it remains unclear to w

UTILMEM: Benchmarking Evidence Utilization in Long-Term Conversational Memory

arXiv:2608.30508v1 Announce Type: new Abstract: Long-term memory is increasingly important for conversational agents, yet existing benchmarks primarily measure memory through pointwise factual recall:

When Does Bigger Help? A Controlled Study of LLM Scale for Ontology Learning

arXiv:2608.31118v1 Announce Type: new Abstract: The effect of Large Language Model (LLM) scale on ontology learning (OL) performance remains insufficiently characterized. We present a controlled evalu

A comprehensive and trustworthy benchmark of AI methods for change detection in Earth observation

arXiv:2608.28247v1 Announce Type: new Abstract: Change detection in Earth observation (EO) is critical for monitoring land surface transformations, yet recent research in the field is constrained by i

A Survey on Rubric-Guided Reinforcement Learning for Language Models

arXiv:2608.27505v1 Announce Type: new Abstract: Reinforcement learning from human feedback (RLHF) has become the dominant paradigm for aligning large language models (LLMs) with human preferences. How

safetyarxiv-cs-cl

Biases in Expected Goals Models Confound Finishing Ability

arXiv:2401.09940v2 Announce Type: replace Abstract: Expected Goals (xG) has emerged as a popular tool for evaluating finishing skill in soccer analytics. It involves comparing a player's cumulative xG

safetyarxiv-cs-lg

CNeo-Bench: Diagnosing Large Language Models on Chinese Neologisms

arXiv:2608.28053v1 Announce Type: new Abstract: Chinese neologisms exploit diverse and unique linguistic mechanisms, such as phonetic substitution (e.g., 886 for ``bye-bye'') and visual character deco

Prompts Without Evidence: How Neuroimaging Mentions Shift Clinical Vision-Language Model Predictions

arXiv:2603.28387v3 Announce Type: replace-cross Abstract: Trustworthy clinical AI must use real evidence and avoid relying on surface-level artifacts. We evaluate 12 open-weight vision-language models

safetyarxiv-cs-lg

Quantization-Triggered Backdoors in Language Models: Cross-Quantizer Transferability and the Validation--Deployment Gap

arXiv:2608.27512v1 Announce Type: cross Abstract: Post-training quantization is often treated as a semantically neutral optimization for edge deployment of Large Language Models. When a full-precision

Quantum SEDONet: Spectrally-Embedded Quantum Deep Operator Networks for Partial Differential Equations

model-releasesarxiv-cs-lg

arXiv:2608.27626v1 Announce Type: cross Abstract: Quantum DeepONet accelerates neural-operator inference by evaluating an orthogonally parameterized network on a quantum computer, reproducing in ideal

XHotpotQA: A Benchmark for Cross-Lingual Knowledge Composition in Multi-Hop Question Answering

arXiv:2608.27481v1 Announce Type: new Abstract: Knowledge-intensive multi-hop question answering requires systems to select evidence and compose dependent facts, yet multilingual benchmarks usually tr

4DSynth: Controllable Procedural World Synthesis for Dynamic Embodied Simulation

arXiv:2608.26947v1 Announce Type: cross Abstract: Embodied agents need environments that are visually diverse, physically interactive, and changing over time. Procedural simulators can generate large

ADeptS-Bench: Measuring the Trustworthiness of Computer Use Agents Across Devices

arXiv:2608.26204v1 Announce Type: cross Abstract: Computer Use Agents (CUAs) are increasingly deployed to navigate mobile and desktop applications on behalf of users, yet no benchmark comprehensively

Ancient-Bench: A Comprehensive Multi-millennial, Multi-medium, and Multi-script Benchmark for Ancient Chinese Artifact Text Recognition

arXiv:2608.27169v1 Announce Type: new Abstract: Ancient Chinese artifact text recognition is fundamental to heritage digitization, and benchmarks for ancient texts are essential for evaluating current

BALMS: Benchmarking Agentic LLMs for Longitudinal Mental Health Sensing

arXiv:2608.27219v1 Announce Type: new Abstract: Mental health assessment relies on episodic self-report scales, which convert subjective states such as stress into numerical scores but provide only sp

Counterfactual Bias Testing for Application Tracking System

arXiv:2608.26899v1 Announce Type: new Abstract: Automated candidate-job matching systems are increasingly classified as high-risk AI under emerging regulation, yet auditing them for demographic bias i

safetyarxiv-cs-ai

FaulT-Bench: Towards Benchmarking Network Troubleshooting LLM Agents under Unreliable User Tickets

arXiv:2608.27021v1 Announce Type: cross Abstract: LLM-based agents are increasingly proposed for network fault diagnosis, but existing benchmarks evaluate them only on accurate tickets and always assu

Improving LLM Interpretability with User-Centric Chain-of-Thought Reasoning

arXiv:2608.26166v1 Announce Type: cross Abstract: Advancing reasoning capabilities allow large language models (LLMs) to tackle increasingly complex problems, while reasoning traces - intermediate ste

Agents

LiveSim: Simulating Environment-Shaped Users in Multi-Agent Live-Stream Ecosystems

arXiv:2608.26849v1 Announce Type: new Abstract: User behavior simulation with large language models~(LLMs) is increasingly used to support multi-agent ecosystem simulation. Existing simulators typical

agentsarxiv-cs-ai

PAWBench: How Far Are We from Probabilistically Aligned World Modeling?

arXiv:2608.27345v1 Announce Type: cross Abstract: Recent video generation models are increasingly framed as world models. Many physical processes can unfold in more than one valid way. Therefore, a wo

Agents

PLCBench: Can Autonomous LLM Agents Turn PLC Access into Sustained Physical Impact?

arXiv:2608.26882v1 Announce Type: cross Abstract: Industrial control systems (ICSs) rely on programmable logic controllers (PLCs) to connect networked computation with physical control. Tool-using lar

agentsarxiv-cs-ai

Representing and Parsing Korean Constituency Structure at Different Levels of Granularity

arXiv:2608.27035v1 Announce Type: new Abstract: Korean constituency parsing raises a representational challenge because the terminal units of a phrase-structure tree do not straightforwardly correspon

researcharxiv-cs-cl

Research Design Tracking and Assessment for the Social Sciences

arXiv:2608.27049v1 Announce Type: new Abstract: Reliable assessment of causal research designs in the social sciences is critical for evidence-based policy-making, yet has so far relied entirely on ma

safetyarxiv-cs-cl

StreamAV-Bench: A Comprehensive Benchmark for Streaming Audio-Video Generation

arXiv:2608.26336v1 Announce Type: cross Abstract: Recent advancements in generative models are pushing video generation toward unbounded streaming audio-video generation for real-time interactive worl

The Cold-Start Safety Gap in LLM Agents

arXiv:2606.07867v2 Announce Type: replace Abstract: Are tool-calling LLM agents equally safe throughout a conversation? We discover they are not: agents are most vulnerable at the very start of a sess

TransMeme: A Multi-Agent Framework for Cross-Cultural Meme Transcreation

arXiv:2608.27127v1 Announce Type: new Abstract: Internet memes are a pervasive form of multimodal online communication; however, such communication often involves users from diverse linguistic and cul

safetyarxiv-cs-ai

A Primer on Computational Semantics for Artificial Intelligence Systems

arXiv:2608.25022v1 Announce Type: new Abstract: As people adopt transformer-based language models (e.g., ChatGPT and Gemini) for an increasing number of use-cases, it is important to know how such mod

A Statistical Audit of Physical AI Benchmark Redundancy

model-releasesarxiv-cs-ro

arXiv:2608.25940v1 Announce Type: new Abstract: Physical AI models are evaluated on suites of benchmarks that differ across model reports, leaving the model-by-benchmark matrix sparse and the relation

Applications

EduDial: Constructing a Large-scale Multi-turn Teacher-Student Dialogue Corpus

arXiv:2510.12899v3 Announce Type: replace Abstract: Recently, several multi-turn dialogue benchmarks have been proposed to evaluate the conversational abilities of large language models (LLMs). As LLM

applicationsarxiv-cs-cl

HealthBench-Psych: A Mental Health Subset of OpenAI's HealthBench

arXiv:2608.25071v1 Announce Type: new Abstract: General-purpose health benchmarks increasingly anchor claims about LLM medical performance, but they are not always resolved by clinical specialty, maki

researcharxiv-cs-cl

Longitudinal Robot Learning from Demonstration with Care Providers in a Home Environment

arXiv:2608.25196v1 Announce Type: new Abstract: Learning from demonstration (LfD) methods enable non-expert end users to teach robots novel skills without explicit programming. However most evaluation

researcharxiv-cs-ro

Mind2Report: Expert-Level Commercial Report Synthesis via Cognitive Deep Research Agent

arXiv:2601.04879v2 Announce Type: replace Abstract: Synthesizing informative commercial reports from massive and noisy web sources is critical for high-stakes business decisions. Although recent deep

Skill Issue: Are Skills Language-Invariant in LLMs?

arXiv:2608.25832v1 Announce Type: new Abstract: Large language models access knowledge inconsistently across languages, but to what extent do they differ in their skill sets when interacting with diff

When agents act on their own, governance has to live in the data layer

Presented by EDB As enterprises give AI agents more autonomy — the ability to plan, decide, and act across systems without a human approving each step — a hard question moves to the center of every ar

safetyventurebeat-ai

ADVERSA: Measuring Multi-Turn Guardrail Degradation and Judge Reliability in Large Language Models

arXiv:2603.10068v2 Announce Type: replace-cross Abstract: Most adversarial evaluations of large language model (LLM) safety assess single prompts and report binary pass/fail outcomes, which fails to c

Diverse by Reasoning: Harnessing the Wisdom of LLM Crowds for Future Prediction

arXiv:2608.24001v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used for future prediction, motivating the use of multiple models as a wisdom-of-the-crowd mechanism. Howe

DoublesEval: Diagnosing Multi-Agent Tactical Reasoning in Vision-Language Models via Professional Doubles Badminton

arXiv:2608.24439v1 Announce Type: new Abstract: Visual Language Models (VLMs) excel at describing visible scene content but struggle to reason about dynamic multi-agent interactions, where action sema

Agents

EgoErrorVQA: Assess Egocentric Comprehension Capabilities through Procedural Errors for Ego-Agentic AI

arXiv:2608.24134v1 Announce Type: new Abstract: The majority of our everyday activities are procedural and consist of sequences of interdependent steps. However, existing benchmarks for Visual Agents

agentsarxiv-cs-cv

Feedback That Backfires: Why Small Language Model Agents Repeat the Call They Just Watched Fail

arXiv:2608.23651v1 Announce Type: cross Abstract: Agent harnesses record a failed tool call and its error message in the transcript and ask the model to continue, on the assumption that the error is c

From Seeing to Acting: Smart Glasses as First-Person Intelligence Platforms

arXiv:2608.24877v1 Announce Type: new Abstract: Smart glasses are evolving from capture and display accessories into first-person intelligence platforms that connect human perception, persistent conte

researcharxiv-cs-cv

Housing Potential Common Data Model and City Digital Twin

arXiv:2605.05535v2 Announce Type: replace Abstract: The evaluation of housing potential requires consideration of a location from multiple perspectives, ranging from zoning and land use to population

Life-Bench: A Benchmark and Knowledge Graph Framework for Multimodal Personalization Beyond Concept Recognition

arXiv:2602.19001v2 Announce Type: replace Abstract: As large language models increasingly power personal assistants, users expect them to reason over multimodal life histories, from recognizing people

MPIB: A Benchmark for Medical Prompt Injection Attacks and Clinical Safety in LLMs

model-releasesarxiv-cs-lg

arXiv:2602.06268v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) systems are increasingly integrated into clinical workflows. However, pr

PatientHub: A Unified Framework for Patient Simulation

arXiv:2602.11684v2 Announce Type: replace-cross Abstract: As Large Language Models increasingly power role-playing applications, simulating patients has become a valuable tool for training counselors

Syn2RealTrack: Bridging the Gap Between Synthetic and Real-World Datasets for Online Multi-View Multi-Target Tracking

arXiv:2608.24130v1 Announce Type: cross Abstract: Multi-camera 3D perception systems for warehouse scenes are trained largely on synthetic data and evaluated on physically captured environments. The r

Taming foundation model with invariance-oriented pre-training for broad-spectrum EEG analysis across signal-level, brain-state, and brain-health tasks

arXiv:2608.24597v1 Announce Type: cross Abstract: Electroencephalography (EEG) is a widely used window into human brain function, but most EEG models remain tied to a one-dataset-one-model supervised