Search: “benchmark”

arXiv:2602.03255v2 Announce Type: replace Abstract: Computer-use agents (CUAs) execute multi-stage tasks through tools, where an early unsafe decision can propagate to consequential actions. Evaluatin

arXiv:2610.02563v1 Announce Type: cross Abstract: We present OpenGameEval, a benchmark and evaluation framework for agentic game development inside Roblox Studio. It runs language models as agents in

arXiv:2610.00969v1 Announce Type: cross Abstract: Large language models (LLMs) have been increasingly used for financial document analysis, including earnings call transcripts (ECTs). Beyond generatin

arXiv:2610.00728v1 Announce Type: cross Abstract: Weather reanalysis products rely on computationally intensive numerical weather predictions followed by data assimilation that corrects the forecast t

arXiv:2505.17613v2 Announce Type: replace Abstract: Automatically evaluating multimodal generation presents a significant challenge, as automated metrics often struggle to align with human evaluation,

arXiv:2610.00817v1 Announce Type: cross Abstract: Join discovery aims to identify tables from large data repositories that can augment a query table with complementary information, enabling downstream

arXiv:2510.23191v2 Announce Type: replace Abstract: Predictive benchmarking, evaluating machine learning models based on predictive performance and competitive ranking, is central to machine learning

arXiv:2610.00118v1 Announce Type: new Abstract: Customer churn prediction on the IBM Telco Customer Churn benchmark (n = 7,043) routinely reports test accuracies above 95%, with the most cited publish

I have been building a somewhat unusual local inference machine around two Huawei Atlas 300I Duo cards. They are relatively inexpensive, passive, dual-accelerator PCIe cards with 96 GB of device memor

arXiv:2609.38652v1 Announce Type: new Abstract: Agentic AI systems increasingly rely on cloud-hosted large language models for planning, tool use, and iterative execution, raising concerns about API c

Evolving MLPerf Training from feature-interaction baselines to production-scale sequential recommendation with HSTU and the 560 GB Yambda-5B embedding benchmark. The post DLRMv4: The Next-Generation R

arXiv:2510.17932v5 Announce Type: replace-cross Abstract: We introduce Chart2Code, a new benchmark for evaluating the chart understanding and code generation capabilities of large multimodal models (L

arXiv:2609.38523v1 Announce Type: cross Abstract: Financial forecasting from earnings conference calls requires models to reason over complex corporate disclosures, market expectations, and subtle com

arXiv:2609.37053v1 Announce Type: new Abstract: Multimodal GUI agents have achieved impressive results on general software benchmarks, yet their ability to operate professional scientific software rem

arXiv:2609.37494v1 Announce Type: new Abstract: Multiple-choice benchmarks are cheap to grade and are running out of room, and the standard remedy, writing harder items, is slow and repeated for every

arXiv:2609.33261v1 Announce Type: new Abstract: Multimodal engineering benchmarks largely evaluate static understanding, such as recognizing components, interpreting diagrams, or answering technical q

arXiv:2609.31651v1 Announce Type: new Abstract: Historical manuscripts remain largely absent from modern vision-language benchmarks, leaving open how well multimodal large language models (MLLMs) hand

arXiv:2509.23219v2 Announce Type: replace Abstract: Technical-domain benchmarks constructed from arXiv papers can overlap the same public text used in LLM pretraining. Auditing this risk at training-c

arXiv:2609.30483v1 Announce Type: cross Abstract: Audio language models state numbers for acoustic quantities, and neither human opinion nor a judge model says whether such a number is true of the sig

arXiv:2608.00008v2 Announce Type: replace Abstract: The local deployment of large language models (LLMs) is gaining traction due to privacy concerns and the desire for on-premise inference. However, t

TL;DR: The Jeff models are a set of Qwen3.5 and Gemma fine-tunes for zero-shot classification: small, efficient, open-weight models with respectable out-of-the-box performance that can be slotted righ

arXiv:2609.30205v1 Announce Type: new Abstract: Large language model (LLM)-based clinical assistants are increasingly being integrated into electronic health record (EHR) systems, transforming how cli

arXiv:2609.27036v1 Announce Type: cross Abstract: Polymer property prediction lacks open, standardized benchmarks that enable rigorous comparison of machine-learning methods, with existing resources c

arXiv:2609.26872v1 Announce Type: new Abstract: Musculoskeletal (MSK) humanoids provide a physiologically grounded embodiment for studying full-body motor control, but their high-dimensional muscle ac

arXiv:2609.26490v1 Announce Type: new Abstract: This study introduces an interdisciplinary framework for benchmarking robots deployed in public environments, addressing the gap between traditional lab

arXiv:2609.26618v1 Announce Type: new Abstract: End-to-end (E2E) driving policies have progressed rapidly on open-loop (OL) benchmarks, yet OL evaluation cannot reveal whether a policy withstands comp

OpenAI: OpenAI releases MentalHealthBench, an open benchmark to evaluate AI responses in realistic mental health conversations, developed with 80+ licensed experts — An open benchmark developed with m

arXiv:2508.15526v2 Announce Type: replace Abstract: The rapid proliferation of large language models (LLMs) has intensified the requirement for reliable safety evaluation to uncover model vulnerabilit

arXiv:2411.10163v3 Announce Type: replace Abstract: Large language models (LLMs) demonstrate remarkable performance across various tasks, prompting researchers to develop diverse evaluation benchmarks

arXiv:2602.16313v2 Announce Type: replace Abstract: Existing evaluations of agents with memory typically assess memorization and action in isolation. One class of benchmarks evaluates memorization by

Small, free finding. I use local Qwen models for typed decisions: a state plus a question with fixed allowed answers, and I read the probability of each answer from the logits of one forward pass inst

arXiv:2609.20832v1 Announce Type: new Abstract: We introduce TatBLiMP, the first benchmark of linguistic minimal pairs for Tatar (tt, ISO 639-3 tat), a Qypchaq Turkic language written in Cyrillic. To

Hey r/LocalLLaMA! I just released PromptDeck v1.1.0 and wanted to share it here since this community is basically the reason it exists. What it is: an open-source desktop studio (Tauri + React) for pr

arXiv:2609.20574v1 Announce Type: new Abstract: Answer grounding in document visual question answering remains an open challenge: most benchmarks lack grounding annotations or provide limited-quality

arXiv:2609.17538v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used for structured information extraction from documents, yet their behavior under realistic OCR noise re

arXiv:2609.18908v1 Announce Type: new Abstract: Existing legal benchmarks cover diverse tasks, while continuous monetary remedies remain comparatively underexplored. We introduce ECtHR-NPD, to the bes

arXiv:2609.17360v1 Announce Type: new Abstract: Full-duplex spoken dialogue systems must distinguish interruptions that require yielding the floor from backchannels that permit continued speaking. Exi

arXiv:2609.16900v1 Announce Type: new Abstract: Platform abuse campaigns conceal redirection instructions with emojis, homophones, character decomposition, and redundant symbols, then route users thro

arXiv:2609.16233v1 Announce Type: cross Abstract: Vision-language models excel at 2D image understanding but remain limited in 3D spatial reasoning. Progress is hindered by limitations in current benc

arXiv:2609.13602v1 Announce Type: new Abstract: Voice agents often need to collect names, addresses, identifiers, dates, and times exactly, yet end-to-end benchmarks obscure where capture fails. We in

arXiv:2511.13725v5 Announce Type: replace-cross Abstract: Malicious AI causing harm to humans is not just a Hollywood fantasy. Indeed, as highly capable models such as Claude Mythos emerge and agent s

arXiv:2607.22365v2 Announce Type: replace Abstract: High-complexity operational environments require methods that characterize temporally distributed patterns rather than classify isolated events. Thi

arXiv:2609.12929v1 Announce Type: new Abstract: Dementia affects an estimated 57 million people worldwide, and for most families the hardest part of care is not memory loss but the behavioral and psyc

arXiv:2609.11185v1 Announce Type: new Abstract: Evidence-based medicine demands strict logical consistency, yet current evaluations of large language models (LLMs) prioritize superficial label matchin

arXiv:2609.11897v1 Announce Type: new Abstract: Causal discovery aims to uncover causal structures from data and is fundamental to scientific reasoning and intervention-based decision making. Its eval

arXiv:2511.13135v3 Announce Type: replace Abstract: Medical vision-language models (VLMs) are increasingly expected to support clinical workflows through diagnostic text and relevant medical images. H