Search: “evaluation”

Are Large Pre-trained Vision Language Models Effective Construction Safety Inspectors?

arXiv:2508.11011v2 Announce Type: replace Abstract: Construction safety inspections typically involve a human inspector identifying safety concerns on-site. With the rise of powerful Vision Language M

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

AssertLLM2: A Comprehensive LLM Benchmark for Assertion Generation from Design Specifications

DGX agent

arXiv:2605.27472v1 Announce Type: cross Abstract: Assertion-based verification (ABV) is a cornerstone of modern hardware design, yet manually translating design intent into formal SystemVerilog Assert

model-releasesarxiv-cs-ai

28 May 2026

Automated Estimation of Impact Time, Impact Location, and Shuttlecock Speed in Badminton Smashes Using Event Cameras

arXiv:2605.28011v1 Announce Type: new Abstract: Quantifying impact phenomena in badminton smashes is important for evaluating both athletic performance and equipment; however, conventional measurement

safetyarxiv-cs-cv

Tutorials

Banger paper from Harvard. AutoScientists drops the central planner entirely. Agents interpret shared experimental data, self-organize aroun…

Banger paper from Harvard. AutoScientists drops the central planner entirely. Agents interpret shared experimental data, self-organize around promising directions, evaluate proposals before resource a

tutorialsdair-ai--x

Beyond pass@k: Redundancy-Aware RLVR for Multi-Sample Code Generation

arXiv:2605.28022v1 Announce Type: new Abstract: LLMs for code generation are commonly evaluated in repeated-sampling settings using Pass@k, where multiple candidate programs are executed against unit

researcharxiv-cs-cl

CALM-IT: Generating Realistic Long-Form Motivational Interviewing Dialogues with Dual-Actor Conversational Dynamics Tracking

arXiv:2601.10085v2 Announce Type: replace Abstract: Therapeutic dialogue is not a sequence of isolated responses: client goals, motivation, resistance, and therapeutic alliance evolve over time. Yet c

researcharxiv-cs-cl

Chinese Word Boundary Recovery through Character Alignment Projection

arXiv:2605.28128v1 Announce Type: new Abstract: Chinese word segmentation is especially fragile in non-standard text, where language learner errors and other character-level divergences disrupt the wo

CircuitLM: A Multi-Agent LLM-Aided Design Framework for Generating Circuit Schematics from Natural Language Prompts

arXiv:2601.04505v3 Announce Type: replace Abstract: Generating accurate circuit schematics from high-level natural language descriptions remains a persistent challenge in electronic design automation

DisasterBench: Benchmarking LLM Planning under Typed Tool Interface Constraints

arXiv:2605.27957v1 Announce Type: new Abstract: Disasters cause severe societal impacts, demanding rapid coordination of heterogeneous AI tools, from satellite analysis to flood prediction and damage

Do LLMs Build World Models From Text? A Multilingual Diagnostic of Spatial Reasoning

arXiv:2605.28277v1 Announce Type: new Abstract: Whether large language models (LLMs) construct internal spatial world models from pure-text descriptions remains contested, and whether such capabilitie

Forget to Know, Remember to Use: Context-Aware Unlearning for Large Language Models

arXiv:2510.17620v2 Announce Type: replace Abstract: Large language models may encode sensitive information or outdated knowledge that needs to be removed, to ensure responsible and compliant model res

researcharxiv-cs-cl

GUI Agents for Continual Game Generation

arXiv:2605.28258v1 Announce Type: cross Abstract: Generating a game is not the same as making one that can be played. Despite advances in code generation, existing approaches treat game generation as

Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows

arXiv:2605.27922v1 Announce Type: new Abstract: LLM agents are increasingly deployed as executable systems that use tools, modify workspaces, and produce concrete artifacts. In such workflows, perform

I signed up for another SaaS

model-releasesben-s-bites

Ben Kamarov reflects on his decision to sign up for yet another SaaS product, likely discussing his evaluation criteria, the specific tool's features, or lessons learned about SaaS adoption and tool p

Let the Results Speak: A Replication-First Paradigm for LLM Behavioral Benchmarking

arXiv:2605.27914v1 Announce Type: cross Abstract: Subjective evaluation of LLM behavior -- empathy, restraint, calibrated emotional tone -- is hard. Human inter-rater agreement on such qualities satur

Measuring Form and Function in Language Models

arXiv:2605.28616v1 Announce Type: cross Abstract: We introduce quantitative metrics for child language acquisition to evaluate language models. Our focus is on the formal syntactic and functional disc

MIRA: A Bilingual Benchmark for Medical Information Response Audit

arXiv:2605.28025v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used to provide public-facing health information, yet existing safety evaluations overlook whether respons

MMTABREAL: Real-World Benchmark for Multimodal Table Understanding

arXiv:2505.21771v2 Announce Type: replace-cross Abstract: Multimodal tables i.e. tabular layouts interleaved with charts, maps, icons, and color encodings are ubiquitous in real applications yet remai

MTAVG-Bench 2.0: Diagnosing Failure Modes of Cinematic Expressiveness in Multi-Talker Audio-Video Generation

arXiv:2605.28035v1 Announce Type: new Abstract: In recent years, Multi-Talker Audio-Video Generation (MTAVG) models have shown promising performance on fundamental metrics such as lip-sync and audio-v

OmniEgo-R^2: A Routed Reasoning Framework for the 1st Cross-Domain EgoCross Challenge at CVPR 2026

arXiv:2605.24481v2 Announce Type: replace Abstract: The 1st Cross-Domain EgoCross Challenge at EgoVis, CVPR 2026 evaluates whether multimodal large language models can reason over egocentric videos ac

researcharxiv-cs-cv

On the Equivariant Learning of the Q-tensor Order Parameter

arXiv:2605.27679v1 Announce Type: cross Abstract: We construct and evaluate group-equivariant neural networks for the prediction of the two-dimensional Q-tensor order parameter of nematic liquid cryst

Operational AI Deployment Assurance: Governance-State Orchestration Under Threshold-Sensitive Deployment Conditions -- A Governance Framework for High-Stakes AI Systems

arXiv:2605.27827v1 Announce Type: new Abstract: AI governance frameworks increasingly emphasize fairness, transparency, accountability, and lifecycle risk management in high-stakes domains. However, m

PEFT-Arena: Understanding Parameter-Efficient Finetuning from a Stability-Plasticity Perspective

arXiv:2605.28819v1 Announce Type: cross Abstract: Parameter-efficient finetuning (PEFT) has become the standard approach for adapting large language models, yet evaluations largely emphasize downstrea

PointQ-Bench: Benchmarking Diagnostic and Interpretable Point Cloud Quality Assessment

arXiv:2605.28241v1 Announce Type: new Abstract: Point cloud quality plays a critical role in 3D acquisition, reconstruction, rendering, and perception, yet existing point cloud quality assessment (PCQ

PortBench: A Correlation-Aware, Full-Pipeline Benchmark for LLM-Driven Portfolio Management

arXiv:2605.27887v1 Announce Type: new Abstract: LLMs have shown strong performance across diverse financial tasks, yet portfolio management (PM), a critical financial decision-making task, remains poo

Applications

Principled Algorithms for Optimizing Generalized Metrics in Multi-Label Learning

arXiv:2605.28767v1 Announce Type: new Abstract: Many real-world classification tasks require predicting multiple labels per instance, necessitating the optimization of complex evaluation metrics such

applicationsarxiv-cs-lg

Securing Retrieval-Augmented Generation: A Taxonomy of Attacks, Defenses, and Future Directions

arXiv:2604.08304v2 Announce Type: replace-cross Abstract: Retrieval-augmented generation (RAG) extends large language models (LLMs) with external knowledge, but this access path also introduces securi

STAB: Specification-driven Testing for Algorithmic Bottlenecks

arXiv:2605.27981v1 Announce Type: new Abstract: Evaluating the efficiency of algorithmic code requires test cases that expose runtime bottlenecks. Previous methods generate efficiency test cases eithe

Structured Belief State and the First Precision-Aware Benchmark for LLM Memory Retrieval

arXiv:2605.11325v2 Announce Type: replace-cross Abstract: Every major benchmark for LLM memory systems, LoCoMo foremost, measures whether a model answered correctly, not whether the memory system retr

The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models

arXiv:2601.19926v2 Announce Type: replace-cross Abstract: We present a systematic review of 337 articles evaluating the syntactic abilities of Transformer-based language models (TLMs), reporting on ov

Thermodynamic properties of chemically disordered compounds via AI-driven estimation of partition function with the PULSE method

arXiv:2605.28594v1 Announce Type: cross Abstract: In this article, we present an improved version of the PULSE method (Partition function Unsupervised Learning Sampling and Evaluation) for estimating

Token Optimization Strategies for LLM-Based Oracle-to-PostgreSQL Migration

arXiv:2605.28557v1 Announce Type: cross Abstract: LLMs are increasingly used for software modernization, code translation, and database migration. However, LLM-based Oracle2PostgreSQL migration remain

Applications

US healthcare still stupidly expensive, with pathetic outcomes, study finds

Although the United States spends more on health care than any other country, it consistently underperforms on key metrics. A Commonwealth Fund study evaluating the U.S. health system relative to 19 o

applicationsars-technica

UserHarness: Harnessing User Minds for Stronger Agent Theory-of-Mind

arXiv:2605.27721v1 Announce Type: cross Abstract: Understanding what a user believes and intends is central to building effective agent assistants. This ability is often evaluated through Theory-of-Mi

Verified Misguidance: Measuring Structural Citation Failures in Search-Augmented LLMs

arXiv:2605.28565v1 Announce Type: cross Abstract: Users of search-augmented LLMs rely on citations as evidence that responses are grounded in real sources, and rarely verify the cited pages themselves

VeriTrip: A Verifiable Benchmark for Travel Planning Agents over Unstructured Web Corpora

arXiv:2605.28683v1 Announce Type: new Abstract: Existing benchmarks have laid the foundation for travel planning agents by establishing API-centric paradigms. However, as the capabilities of Autonomou

When Think-with-Image Meets Safety: What Determines Multimodal Jailbreak Robustness?

arXiv:2605.27932v1 Announce Type: cross Abstract: Think-with-image reasoning is emerging as a new inference paradigm for large vision-language models, but its safety implications remain poorly underst

Local Ai

AIDG: A Formal Decomposition of Information Extraction and Containment Asymmetries in Multi-Turn LLM Dialogue

arXiv:2602.17443v2 Announce Type: replace Abstract: Multi-turn LLM evaluation is typically reported as a single win-rate scalar, conflating distinct capabilities. We introduce AIDG (Adversarial Inform

local-aiarxiv-cs-cl

Benchmark Leakage Trap: Can We Trust LLM-based Recommendation?

model-releasesarxiv-cs-lg

arXiv:2602.13626v3 Announce Type: replace Abstract: The expanding integration of Large Language Models (LLMs) into recommender systems poses critical challenges to evaluation reliability. This paper i

Beyond Holistic Models: Systematic Component-level Benchmarking of Deep Multivariate Time-Series Forecasting

model-releasesarxiv-cs-lg

arXiv:2605.26562v1 Announce Type: new Abstract: While previous research in multivariate time series forecasting has focused on developing complex holistic models, this work advocates for a shift towar

ChartAct: A Benchmark for Dynamic Chart Understanding

arXiv:2605.26994v1 Announce Type: new Abstract: Charts are widely used to present complex data for analysis and decision making. Existing chart understanding benchmarks mainly focus on static charts,

Composition Collapse: Stable Factual Knowledge Does Not Imply Compositional Reasoning

arXiv:2605.26789v1 Announce Type: new Abstract: Post-training is routinely evaluated through aggregate benchmark scores that treat multi-hop reasoning as a single capability -- as if a model that answ

Counterfactual Credit Policy Optimization for Multi-Agent Collaboration

arXiv:2603.21563v2 Announce Type: replace Abstract: Collaborative multi-agent large language models (LLMs) can solve complex reasoning tasks by decomposing roles, but reinforcement learning for such s

Drive-P2D: A Progressive Perception-to-Decision Benchmark for VLMs in Autonomous Driving

arXiv:2601.14702v2 Announce Type: replace Abstract: Autonomous driving requires reliable perception and safe decision-making in complex scenarios. Recent vision-language models (VLMs) demonstrate reas

EdgeFlow: Edge-Map Augmented VLM-Based Flowchart Processing for Industrial Requirements Engineering

arXiv:2605.27332v1 Announce Type: cross Abstract: Flowcharts are widely used in industrial requirements, but usually remain embedded as static images. Vision Language Models (VLMs) show promise in the

HTMLCure: Turning Browser Experience into State Guided Repair for Interactive HTML

arXiv:2605.26807v1 Announce Type: cross Abstract: LLMs can now produce full HTML pages, but many of those pages are only superficially correct: they render once, then fail under scroll, hover, click,

Improving your agent has been a manual process of: ✅ Reading traces ✅ Looking for patterns ✅ Writing evals ✅ Creating fixes Now, LangSmith E…

LangSmith has introduced automated tools to streamline agent improvement, eliminating the manual workflow of reading execution traces, identifying patterns, writing evaluations, and implementing fixes

agentsharrison-chase--x

InfoSynth: Information-Guided Benchmark Synthesis for LLMs

arXiv:2601.00575v2 Announce Type: replace Abstract: Large language models (LLMs) have demonstrated significant advancements in reasoning and code generation, but efficiently creating new benchmarks to