Search: “evaluation”

arXiv:2609.01548v1 Announce Type: new Abstract: Large Language Models (LLMs) are increasingly used in advice seeking and decision making that may affect social judgements. Despite stigma's profound ef

Banger paper from the Qwen team. If you evaluate agents on anything longer than a single session, this one is worth your time. (bookmark it) E-Commerce Bench runs an agent through a simulated 365-day

arXiv:2608.30536v1 Announce Type: new Abstract: Reliable execution of long-horizon mobile manipulation tasks remains challenging because overall task success depends on the successful completion of mu

arXiv:2608.30047v1 Announce Type: new Abstract: Recent AI systems promise autonomous scientific discovery, claiming to discover algorithms and produce research papers, yet understanding whether they e

arXiv:2608.29929v1 Announce Type: new Abstract: Vehicle attribute recognition is an important task in intelligent transportation systems, particularly when Automatic License Plate Recognition (ALPR) i

arXiv:2608.29543v1 Announce Type: cross Abstract: Recent advances in large language models (LLMs) have established conversational text-to-SQL as a practical interface between users and databases, ofte

arXiv:2608.29738v1 Announce Type: new Abstract: Large language models (LLMs) can generate apparently highly persuasive text, but does sounding persuasive mean arguing well? We introduce extsc{Persuasi

arXiv:2512.19161v2 Announce Type: replace Abstract: Subtitles are essential for video accessibility and audience engagement. Modern Automatic Speech Recognition (ASR) systems, built upon Encoder-Decod

arXiv:2608.31057v1 Announce Type: new Abstract: Agent working memory is heterogeneous. Objects such as instructions, artifacts, tool outputs, and agent-generated state play different semantic roles an

arXiv:2608.30856v1 Announce Type: new Abstract: Refusals are often treated as face-threatening acts in pragmatics because they can challenge the requester's socially claimed self-image. Large language

arXiv:2608.27756v1 Announce Type: new Abstract: Determining whether large language models derive answers from context or prior knowledge remains a fundamental challenge. Self-contained linguistic olym

arXiv:2608.26119v1 Announce Type: new Abstract: Whether large language models can be prompted to generate rhetorical fallacies on demand, and whether current safety post-training constrains this behav

arXiv:2608.25529v1 Announce Type: new Abstract: Multimodal Large Language Models (MLLMs) have shown strong performance in video understanding. However, their ability to follow instructions in this dom

arXiv:2608.23164v1 Announce Type: cross Abstract: Counterfactual (CF) explanations for time-series classifiers are usually evaluated one example at a time: what minimal edit flips this single window's

arXiv:2608.22770v1 Announce Type: new Abstract: Financial institutions need an independent way to detect missing, stale, and misclassified corporate-event records in vendor databases. We introduce Sea

arXiv:2608.22652v1 Announce Type: cross Abstract: LLMs are increasingly used for code generation, yet they frequently hallucinate non-existent software packages, creating exploitable entry points into

arXiv:2407.20360v2 Announce Type: replace Abstract: Digital Simulations (DS) provide safe environments where users interact with an agent through conversational prompts, providing engaging learning ex

arXiv:2402.14879v2 Announce Type: replace-cross Abstract: To enhance immersion and engagement in video games, the design of Affective Non-Player Characters (ANPCs) is a key focus for researchers and p

Katie Roof / Business Insider: Sources: Hugging Face is exploring a sale that could value it at 13B+, up from 4.5B in 2023, and has been working with a bank to evaluate bidders' interest — The AI indu

arXiv:2608.19974v1 Announce Type: new Abstract: LLM agents in financial markets may cite rules yet still submit orders that violate executable constraints or misread surveillance evidence. We introduc

arXiv:2608.18539v1 Announce Type: cross Abstract: The remarkable capabilities of large language models (LLMs) are often undermined by their instability. Even subtle and semantically irrelevant changes

arXiv:2512.14629v2 Announce Type: replace-cross Abstract: Music editing plays a vital role in modern music production, with applications in film, broadcasting, and game development. Recent advances in

arXiv:2510.03115v2 Announce Type: replace Abstract: Speech-to-Text Translation (S2TT) systems built from Automatic Speech Recognition (ASR) and Text-to-Text Translation (T2TT) modules face two major l

Modern AI agents increasingly rely on search infrastructure to execute complex, neuro-symbolic reasoning workflows. These workflows often compile into deeply nested, non-monotonic Boolean queries over

arXiv:2608.14808v1 Announce Type: new Abstract: When a user question is underspecified, a capable model should recognize that its context is insufficient, identify the missing information, ask for it,

arXiv:2608.14863v1 Announce Type: cross Abstract: LLM-based coding agents have advanced rapidly on single-process SWE tasks, with frontier models now clustering in the high-70s on SWE-bench Verified.

arXiv:2608.15446v1 Announce Type: new Abstract: This paper presents an evaluation of a goal-free probabilistic framework for human intent inference during robotic manipulation. We deploy the Global Us

arXiv:2608.16081v1 Announce Type: new Abstract: Open-weight and frontier vision-language models (VLMs) perform well on general image understanding, but their ability to interpret fine-grained hand ges

arXiv:2505.07372v3 Announce Type: replace-cross Abstract: This paper presents a novel methodology for enhancing Automated Program Repair (APR) through synthetic data generation utilizing Large Languag

arXiv:2608.14886v1 Announce Type: new Abstract: Retrieval quality in RAG systems is commonly reported as a single aggregate score, which can hide large differences across query types and language cond

arXiv:2608.16747v1 Announce Type: cross Abstract: Many areas of AI research, such as language model interpretability and chain of thought faithfulness, seek to explain model behaviors. But what consti

arXiv:2608.11341v1 Announce Type: new Abstract: Apollo did not reach the Moon merely because its engineers could solve difficult equations. It succeeded by turning a distant ambition into a mission ar

arXiv:2608.11942v1 Announce Type: new Abstract: Quantitative MRI (qMRI) provides standardised tissue parameter maps, but the reliability of deep learning-based qMRI mapping methods is often not explic

arXiv:2608.11242v1 Announce Type: cross Abstract: When the context window is under pressure, LLM systems compact prior context to continue ongoing tasks. We identify a class of user-issued instruction

arXiv:2608.10475v1 Announce Type: new Abstract: The emergence of language-based AI agents promises to transform the scope of machine economic activity. Instead of just proposing bids or following hard

arXiv:2608.10692v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly deployed as mobile assistants, where a key challenge is leveraging personal information scattered across

arXiv:2608.09296v1 Announce Type: new Abstract: A CAD model is not engineering-grade merely because it looks correct. It must satisfy design requirements, respond predictably to parameter changes, sup

arXiv:2608.07688v1 Announce Type: new Abstract: IT audits require auditors to judge whether heterogeneous organizational evidence satisfies semantic security and compliance controls. This judgment is

arXiv:2510.22087v3 Announce Type: replace-cross Abstract: The field of computer architecture, which bridges high-level software abstractions and low-level hardware implementations, remains absent from

arXiv:2608.07473v1 Announce Type: new Abstract: Evaluative AI (EAI) has been recently proposed as a way to support human decision-making, not by producing a single recommendation, but by presenting co

arXiv:2608.06867v1 Announce Type: new Abstract: No single large language model (LLM) is optimal across all queries and budget constraints, making model routing essential for cost-effective deployment.

I'm a retired platform engineer, coding mainly in Rust, and involved with a ML study group. I developed a ML programming language (alternative to Python, Colab) to help me learn (and teach) ML concept

arXiv:2602.12187v2 Announce Type: replace-cross Abstract: Search-Augmented Generative Engines (SAGE) have emerged as a new paradigm for information access, bridging web-scale retrieval with generative

When you move a model into production, you want the quality you evaluated to carry through the serving stack. @Kimi_Moonshot benchmarked Kimi K3 across major inference providers, and Together AI ranke

After evaluating one of our upcoming models, Astra, we're treating it as our first 'critical' model for cybersecurity under our Preparedness Framework. This is a scenario we've planned for, and we're

arXiv:2608.02616v1 Announce Type: cross Abstract: We present the first independent, systematic evaluation of OpenAI's Privacy Filter (OPF), a 1.5B-parameter bidirectional PII detector, across 42 synth