Search: “evaluation”

NumBench: Diagnosing Counting Failures in Text-to-Image Models

arXiv:2608.28206v1 Announce Type: new Abstract: Text-to-image (T2I) models often generate the wrong number of objects, yet existing benchmarks are too small or weakly controlled to explain why. We int

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Agents

One year in a forest: Analyzing the challenges of autonomous navigation in subarctic environments

DGX agent

arXiv:2608.27628v1 Announce Type: new Abstract: Subarctic regions have the potential to see increased deployment of autonomous robots in applications including forestry, mining, and environmental moni

agentsarxiv-cs-ro

31 Aug 2026

PersonaForge: Realistic Multi-Turn User Simulation for Agentic Systems

arXiv:2608.28378v1 Announce Type: new Abstract: Large language models are increasingly used as agentic workflow executors, yet existing training data and benchmarks largely assume informationally comp

QUORUM: QUality-Optimized Routing Using Multiple annotators

arXiv:2608.27974v1 Announce Type: new Abstract: Data annotation remains a central bottleneck in natural language processing, requiring human effort to obtain high-quality labels at scale. While Large

Real-Time Musculoskeletal Surrogates for Pediatric Cerebral Palsy: a Credibility Pilot

arXiv:2608.28371v1 Announce Type: new Abstract: Real-time musculoskeletal (MSK) surrogates could support personalized rehabilitation for children with cerebral palsy (CP), but their credibility depend

researcharxiv-cs-cv

Remote Human and Robot Interaction for Greenhouse Gardening Using Virtual Reality

arXiv:2608.27545v1 Announce Type: new Abstract: This study evaluates the effectiveness of remote human-robot interaction using virtual reality for leaf inspection and soil moisture assessment in a gre

researcharxiv-cs-ro

REPLICANT: Learning Policies for Evading and Hardening Malware Detectors

arXiv:2608.28499v1 Announce Type: new Abstract: To determine the real-world effectiveness of machine learning based malware detection, it is vital to evaluate its robustness against highly capable adv

safetyarxiv-cs-lg

Retrieving Relations, Detecting Fallacies: A RAG Approach to Political Debate Analysis

arXiv:2608.27471v1 Announce Type: cross Abstract: Fallacies are arguments that employ invalid reasoning, making their automatic detection critical in sensitive contexts such as high-stakes political d

Roleplaying with Structure: Synthetic Therapist-Client Conversation Generation from Questionnaires

arXiv:2510.25384v2 Announce Type: replace Abstract: Large Language Models (LLMs) are promising tools for synthetic data generation in mental health. However, privacy policies and restrictions forced p

Steering Multimodal Large Language Models Decoding for Context-Aware Safety

arXiv:2509.19212v2 Announce Type: replace Abstract: Multimodal Large Language Models (MLLMs) are increasingly deployed in real-world applications, yet their ability to make context-aware safety decisi

Synth-JDoc: Synthesizing a Japanese Document Image Dataset for OCR with Diverse Layouts and Embedded Images

arXiv:2608.28248v1 Announce Type: cross Abstract: The ability of Large Vision Language Models (LVLMs) to read text within document images is crucial, as it enables various applications such as Documen

Agents

What Makes Agent Memory Useful for Reliable Unanswerable Question Handling?

arXiv:2608.27924v1 Announce Type: new Abstract: Reliable handling of unanswerable questions (UAQs) is critical for trustworthy LLM-based agents. Although memory is widely used in agent systems, its ro

agentsarxiv-cs-cl

When Robots Mishear Us: Mapping the Safety Risks of Voice-Controlled Embodied AI

arXiv:2608.28518v1 Announce Type: cross Abstract: We investigate whether automatic speech recognition (ASR) errors in user input can lead to unsafe outputs from Embodied AI (EAI) models. We find that

A Task-Centric Ontology and Deterministic Domain Rules as a Verifiable Core for AI-Assisted Chemistry Problem Solving

arXiv:2608.26164v1 Announce Type: new Abstract: Large language models can interpret natural-language chemistry questions, but their internal reasoning is difficult to inspect, constrain, and validate.

researcharxiv-cs-ai

Accelerating Scientific Research with Gemini in the Real-World

arXiv:2608.26701v1 Announce Type: new Abstract: We present an extension and comprehensive real-world validation of Co-Scientist, a Gemini-based multi-agent system designed to accelerate end-to-end sci

AEScorer: An Agentic Evidence-Grounded Framework for Graded Factuality Verification

arXiv:2601.03605v2 Announce Type: replace Abstract: Despite the significant advancements of Large Language Models (LLMs), their factuality remains a critical challenge, creating a growing need for mor

Autoresearch with Coding Agents: Generalizers and Metric-Maximizers on Quran Recitation Data

arXiv:2607.18064v2 Announce Type: replace-cross Abstract: Coding agents can now be left alone to improve software against a score. In this pattern--recently popularized as 'autoresearch'--the agent re

BekchiAI: Measuring, Observing, and Controlling LLM Agents in One Click

arXiv:2608.26867v1 Announce Type: new Abstract: Large language model agents reason, call tools, and act autonomously over many steps, but their agentic skills-correctly sequencing tools, planning unde

Benchmarking AI Agents for Hardware Design Automation via MCP Tool Calling

arXiv:2608.26199v1 Announce Type: new Abstract: We ask whether AI agents powered by locally deployed large language models can reliably automate expert-defined hardware design workflows in an industry

BrailleBench: Investigating Multi-Criteria Braille Comprehension in Large Language Models

arXiv:2608.27268v1 Announce Type: new Abstract: Although Large language models (LLMs) mediate access to knowledge and computational assistance, their capabilities should benefit vulnerable groups in t

Clinically Aligned Geometry Constraints for Robust IVUS Vessel Boundary Segmentation

arXiv:2606.18723v2 Announce Type: replace Abstract: Intravascular ultrasound (IVUS) lumen and external elastic membrane (EEM) segmentation is important for quantitative coronary plaque burden assessme

researcharxiv-cs-cv

Comparing Chunking and Embedding Strategies for Turkish RAG Systems

arXiv:2608.26192v1 Announce Type: cross Abstract: How documents are segmented into retrievable chunks and how those chunks are embedded strongly affect Retrieval-Augmented Generation (RAG) quality, ye

researcharxiv-cs-ai

Compositional Generalization via Structural Identification in a Category-Theoretic Framework

arXiv:2608.26465v1 Announce Type: new Abstract: Compositional generalization is usually evaluated through model accuracy. We instead ask which structural or lexical identifications make held-out COGS

DocTalkBN: A Novel Dataset of Expert Telemedicine Conversations in Bengali

arXiv:2608.27110v1 Announce Type: new Abstract: Reliable medical conversational AI requires authentic expert--patient interaction data, yet such datasets remain scarce, especially for low-resource lan

Don't Overthink, Don't Underthink: Toward Adaptive Reasoning in Agentic AI

arXiv:2608.26442v1 Announce Type: new Abstract: Recent advances in Large Language Models (LLMs) have shown that increased inference-time reasoning can improve performance on complex tasks. However, ma

DRL: A Deterministic Relational Middleware Layer for Transaction-Safe Enterprise NL2SQL Under Schema-Graph Scaling

arXiv:2608.26172v1 Announce Type: cross Abstract: Deploying natural-language interfaces over enterprise OLTP catalogs fails at scale because semantic parsers collapse under schema-graph scaling, infla

FireRedAudio: A General-Purpose Audio Language Model with Decoupled Continuous Representations for Understanding and Generation

arXiv:2608.24168v2 Announce Type: replace Abstract: A unified audio model must recognize and understand linguistic, paralinguistic, and environmental information while supporting speech synthesis and

Agents

Knowing When Not to Reuse: Conditional Experience Transfer in Autonomous LLM Post-Training

arXiv:2608.26730v1 Announce Type: new Abstract: Large language models offer broad capabilities, but adapting them to evolving domains, tools, and requirements often entails repeated post-training. Aut

agentsarxiv-cs-ai

LLMs Can Design Near-Optimal OR Algorithms

arXiv:2608.27296v1 Announce Type: new Abstract: We ask whether large language models (LLMs) can design effective algorithms for well-specified operations research (OR) problems. We study inventory con

Multi2AV-Safety: Benchmarking Safety in Multimodal-to-Audio-Video Generation

arXiv:2608.26535v1 Announce Type: new Abstract: Audio-video generation is rapidly moving from prompt-driven synthesis toward multimodal conditioning, where text, images, audio, and video can jointly s

PragAlign: Evidence-Sensitive Reply Assistance Across Chinese and Japanese Appropriateness Judgments

arXiv:2608.26700v1 Announce Type: new Abstract: Reply assistance in multilingual settings requires linguistic competence and culturally situated judgments of appropriateness. We present PragAlign, whi

Prediction of Prediction (PoP): Inter-Layer Activation Fusion for Single-Pass Hallucination Detection in Large Language Models

arXiv:2608.27165v1 Announce Type: new Abstract: Autoregressive large language models (LLMs) routinely generate factually incorrect outputs with high decoding confidence, limiting their deployment in h

RATIO: A Benchmark for Retrieval Across Typed Ideation Operations in Scientific Literature

arXiv:2608.27394v1 Announce Type: new Abstract: Retrieved scientific literature can serve as inspiration for both human and AI scientists. Inspiration can take different forms: prior work may directly

Residual Deep Reinforcement Learning-Based Computed Torque Control for a Cable-Driven Lower-Limb Rehabilitation Robot under Disturbances and Parametric Uncertainties

arXiv:2608.26739v1 Announce Type: new Abstract: Accurate trajectory tracking in cable-driven lower-limb rehabilitation robots is challenging because model uncertainty, external disturbances, joint con

safetyarxiv-cs-ro

RuleWeaver: Benchmarking Rule-Centered Scenario Reasoning for Large Language Models

arXiv:2608.26832v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly applied to specialized domains, where effective use of domain expertise often requires reasoning over comp

Agents

SKILL.state: Scalable Long-Horizon Agent Skills

arXiv:2608.26263v1 Announce Type: new Abstract: Large Language Models (LLMs) increasingly act as autonomous agents executing complex, long-running procedural skills. Existing agent runtimes maintain e

agentsarxiv-cs-ai

Temporally-Grounded Language Generation: Towards Real-Time Vision-Language Models

arXiv:2505.11326v2 Announce Type: replace-cross Abstract: Vision-language models (VLMs) have shown remarkable progress in offline tasks such as image captioning and video question answering. However,

What the 'Spotless' Mind Remembers: How Knowledge Entanglement Shapes What Leaks After Unlearning in LLMs

arXiv:2510.25732v2 Announce Type: replace-cross Abstract: Unlearning in large language models (LLMs) is usually evaluated as whether an 'unlearned' fact can be recovered. We instead ask whether a fact

researcharxiv-cs-ai

A General-Purpose Molecular Foundation Model Transfers Across Diverse Olfactory Tasks

model-releasesarxiv-cs-lg

arXiv:2608.25893v1 Announce Type: new Abstract: Foundation models have transformed molecular property prediction, yet it remains unclear whether a molecular foundation model, fine-tuned on a single ca

A Layered Security Framework Against Prompt Injection in RAG-Based Chatbots

arXiv:2606.19660v2 Announce Type: replace-cross Abstract: Prompt injection is ranked as the most critical vulnerability in large language model (LLM) deployments by the OWASP Top 10 for LLM Applicatio

AutoVerifier: Residual-Guided Non-Parametric Optimization for Reference-Based Answer Verification

arXiv:2608.25637v1 Announce Type: new Abstract: Reference-based verifiers are important for evaluating reasoning models and providing accurate outcome rewards in reinforcement learning with verifiable

Can You Trust Frozen Hematology Foundation Models under Acquisition Shift?

arXiv:2608.25148v1 Announce Type: new Abstract: Frozen hematology foundation-model (FM) embeddings reach near-saturated in-domain white-blood-cell (WBC) accuracy, but clinical deployment demands relia

CEDAR: Controlled and Event-Driven Demand Forecasting via Residual Decomposition

arXiv:2608.25871v1 Announce Type: new Abstract: Forecasting in large-scale e-commerce marketplaces is increasingly required to support planning: merchants need to evaluate sales outcomes under future

safetyarxiv-cs-lg

CropCop: An Auditable 120-Class Plant-Health Model from Benchmark Reconstruction to a Quantised Runtime Artifact

arXiv:2608.25539v1 Announce Type: new Abstract: A plant-health score can appear precise while resting on duplicated image families, a long-tailed label space, or a runtime file that was never evaluate

Flower Hub: A Reproducible Benchmarking Platform for Federated Learning in Simulation and Deployment

model-releasesarxiv-cs-lg

arXiv:2608.25114v1 Announce Type: new Abstract: Federated learning (FL) has emerged as a key approach for training models across decentralized data, yet benchmarking in FL remains difficult to reprodu

Formal, Executable and Explainable Runtime Monitoring of Spoken Air Traffic Control Operational Procedures

arXiv:2608.25926v1 Announce Type: cross Abstract: Air traffic control procedures are executed through spoken exchanges between controllers and pilots. These interactions are essential to the safety of

From National Curricula to Cultural Awareness: Constructing Open-Ended Culture-Specific Question Answering Dataset

arXiv:2601.04632v2 Announce Type: replace Abstract: Large language models (LLMs) achieve strong performance on many tasks, but their progress remains uneven across languages and cultures, often reflec

GTPred: Benchmarking MLLMs for Interpretable Geo-localization and Time-of-capture Prediction

arXiv:2601.13207v2 Announce Type: replace Abstract: Geo-localization aims to infer the geographic location where an image was captured using observable visual evidence. Traditional methods achieve imp