Search: “evaluation”

arXiv:2410.04509v3 Announce Type: replace Abstract: As the field of Multimodal Large Language Models (MLLMs) continues to evolve, their potential to revolutionize artificial intelligence is particular

arXiv:2604.16862v1 Announce Type: new Abstract: Recent deployments of large language models (LLMs) as autonomous trading agents raise questions about whether financial decision-making competence gener

arXiv:2604.17650v1 Announce Type: new Abstract: LLMs are increasingly deployed in dynamic, real-world settings, where the distribution of user prompts can shift substantially over time as new tasks, p

arXiv:2601.04029v2 Announce Type: replace Abstract: Large Audio-Language Models (LALMs) as judges have emerged as a prominent approach for evaluating speech generation quality, yet their ability to as

arXiv:2604.16256v1 Announce Type: cross Abstract: Reasoning in vision-language models (VLMs) has recently attracted significant attention due to its broad applicability across diverse downstream tasks

arXiv:2604.15882v1 Announce Type: cross Abstract: We introduce JFinTEB, the first comprehensive benchmark specifically designed for evaluating Japanese financial text embeddings. Existing embedding be

arXiv:2604.16022v1 Announce Type: new Abstract: As Large Language Models (LLMs) transition from text processors to autonomous agents, evaluating their social reasoning in embodied multi-agent settings

arXiv:2604.06647v2 Announce Type: replace Abstract: Retrieval-Augmented Generation (RAG) systems are typically evaluated under static assumptions, despite being frequently corrected through user or ex

arXiv:2604.08064v2 Announce Type: replace Abstract: Existing memory benchmarks for LLM agents evaluate explicit recall of facts, yet overlook implicit memory where experience becomes automated behavio

arXiv:2509.06477v2 Announce Type: replace Abstract: Shortcuts such as APIs and deep-links have emerged as efficient complements to flexible GUI operations, fostering a promising hybrid paradigm for ML

arXiv:2505.19662v3 Announce Type: replace-cross Abstract: This paper introduces FieldWorkArena, a benchmark for agentic AI targeting real-world field work. With the recent increase in demand for agent

arXiv:2604.13201v1 Announce Type: new Abstract: Large language models are emerging as scientific assistants, but evaluating their ability to reason from empirical data remains challenging. Benchmarks

arXiv:2511.11334v3 Announce Type: replace Abstract: The rapid advancement of large language models (LLMs) has not been matched by their evaluation in low-resource languages, especially Southeast Asian

arXiv:2604.13072v1 Announce Type: new Abstract: LLM-based agents are increasingly expected to handle real-world assistant tasks, yet existing benchmarks typically evaluate them under isolated sources

arXiv:2604.13828v1 Announce Type: new Abstract: User simulators are essential for the scalable training and evaluation of interactive AI systems. However, existing approaches often rely on shallow use

arXiv:2507.14005v2 Announce Type: replace Abstract: It was recently shown that dynamic programming (DP) methods for finding static CVaR-optimal policies in Markov Decision Processes (MDPs) can fail wh

arXiv:2604.14041v1 Announce Type: new Abstract: Daily scenarios are characterized by visual richness, requiring Multimodal Large Language Models (MLLMs) to filter noise and identify decisive visual cl

arXiv:2604.12875v1 Announce Type: new Abstract: The rapid expansion of large language model (LLM) safety evaluation has produced a substantial benchmark ecosystem, but not a correspondingly coherent m

arXiv:2604.12659v1 Announce Type: cross Abstract: Vision-language models(VLMs) are increasingly applied to visual stock price forecasting, yet existing benchmarks inadequately evaluate their understan

arXiv:2503.21380v3 Announce Type: replace Abstract: The rapid advancement of large reasoning models has saturated existing math benchmarks, underscoring the urgent need for more challenging evaluation

arXiv:2604.10503v1 Announce Type: cross Abstract: Modern audio systems universally employ mel-scale representations derived from 1940s Western psychoacoustic studies, potentially encoding cultural bia

arXiv:2512.18073v2 Announce Type: replace Abstract: Multimodal LLMs (MLLMs) are capable of performing complex data analysis, visual question answering, generation, and reasoning tasks. However, their

arXiv:2604.11307v1 Announce Type: new Abstract: Leveraging Multi-modal Large Language Models (MLLMs) to accelerate frontier scientific research is promising, yet how to rigorously evaluate such system

arXiv:2604.09870v1 Announce Type: cross Abstract: We investigate how looped transformers encode human preference in their internal iteration states. Using Ouro-2.6B-Thinking, a 2.6B-parameter looped t

arXiv:2604.10720v1 Announce Type: new Abstract: Artificial models that simulate how learners act and respond within educational systems are a promising tool for evaluating tutoring strategies and feed

arXiv:2604.08966v1 Announce Type: new Abstract: While Multimodal Large Language Models (MLLMs) have advanced Video Temporal Grounding (VTG), existing methods often couple output paradigms with differe

This r/MachineLearning post discusses the landscape of frameworks and tools used to support benchmarking of LLMs and agentic AI systems, covering how to systematically evaluate model capabilities beyo

arXiv:2604.06566v1 Announce Type: cross Abstract: As the complexity of modern workloads and hardware increasingly outpaces human research and engineering capacity, existing methods for database perfor

arXiv:2603.18472v2 Announce Type: replace-cross Abstract: Multimodal large language models (MLLMs) perform strongly on natural images, yet their ability to understand discrete visual symbols remains u

More organizations are using natural language to query data instead of writing manual SQL. But moving an AI agent from a prototype to a production-ready tool requires rigorous, repeatable testing. Pri

arXiv:2503.08292v4 Announce Type: replace-cross Abstract: Large language models (LLMs) are increasingly applied to outpatient referral tasks across healthcare systems. However, there is a lack of stan

arXiv:2602.22683v2 Announce Type: replace Abstract: The rapid advancement of AI-powered smart glasses-one of the hottest wearable devices-has unlocked new frontiers for multimodal interaction, with Vi

arXiv:2610.21799v1 Announce Type: new Abstract: Machine unlearning aims to remove targeted information from large language models, but existing evaluations often measure forgetting only on a designate

arXiv:2610.21525v1 Announce Type: new Abstract: Large Language Models (LLMs) have shown impressive capabilities across many tasks, including processing large amounts of text, following instructions, a

arXiv:2610.21597v1 Announce Type: cross Abstract: Reverse engineering the data formats used by mobile software is skilled and slow, and using AI to assist with it is attractive. We evaluate this for t

arXiv:2610.21568v1 Announce Type: new Abstract: Recent multimodal large language models have advanced video understanding, but existing benchmarks mainly evaluate whether models can interpret observed

arXiv:2610.20776v1 Announce Type: cross Abstract: LLM-based agents increasingly execute long-horizon, end-to-end workflows involving shell commands, code modification, dependency installation, and int

arXiv:2610.21591v1 Announce Type: new Abstract: Machine learning (ML) for analog IC design is constrained by limited open data and the absence of standardized evaluation: results reported in different

arXiv:2610.22155v1 Announce Type: new Abstract: Real-time interaction requires agents to perceive evolving environments and act under temporal constraints, yet existing evaluations of real-time stream

arXiv:2610.21438v1 Announce Type: new Abstract: Existing deepfake detection models are built upon manipu- lated images that are tampered with either exclusively fully swapped or fully synthesised face

arXiv:2610.17710v1 Announce Type: new Abstract: Training graphical user interface (GUI) agents with reinforcement learning suffers from sparse outcome rewards, leading to inefficient learning. Process

arXiv:2610.18498v1 Announce Type: cross Abstract: Autoformalization (AF) requires more than compilation: formal statements must preserve the meaning of their informal sources, a property known as fait

arXiv:2505.21710v2 Announce Type: replace Abstract: This study evaluates the effectiveness of ChatGPT, an advanced AI model for natural language processing, in identifying targeting and inappropriate

arXiv:2606.17029v2 Announce Type: replace Abstract: Deep research agents synthesize long-form reports by searching and reasoning over retrieved evidence. Reinforcement learning with rubric-based rewar

arXiv:2610.20598v1 Announce Type: new Abstract: Achieving general spatial intelligence requires a unified understanding of both static 3D and dynamic 4D. However, we find that existing models struggle

arXiv:2603.13535v5 Announce Type: cross Abstract: Evaluating Ollivier-Ricci (OR) curvature on large-scale graphs is computationally prohibitive due to the necessity of solving an optimal transport pro