Search: “benchmark”

arXiv:2510.18368v2 Announce Type: replace Abstract: We present extbf{Korean SimpleQA (KoSimpleQA)}, a benchmark for evaluating factuality in large language models (LLMs) with a focus on Korean cultura

arXiv:2601.15645v2 Announce Type: replace Abstract: Large-scale language models (LLMs) often offer clinical judgments based on incomplete information, increasing the risk of misdiagnosis. Existing stu

arXiv:2510.13852v3 Announce Type: replace-cross Abstract: Is an LLM telling you different facts than it's telling me? This paper introduces ConsistencyAI, an independent benchmark for measuring the fa

arXiv:2609.08149v1 Announce Type: new Abstract: SWE-Bench Pro has emerged as a standard benchmark for evaluating software engineering agents on challenging repository-level tasks. However, our analysi

arXiv:2609.05902v1 Announce Type: new Abstract: Recent image generation systems increasingly combine multimodal understanding, reasoning, and synthesis, suggesting that they may do more than render pl

arXiv:2602.00288v4 Announce Type: replace Abstract: Fine-grained spatio-temporal understanding is essential for video reasoning and embodied AI. Yet, while Multimodal Large Language Models (MLLMs) mas

arXiv:2609.06245v1 Announce Type: new Abstract: Multimodal Large Language Models (MLLMs) perform strongly on general visual understanding tasks such as visual question answering, yet they often strugg

We're introducing Q2D-Web (Query2Doc-Web), a benchmark and public leaderboard for evaluating retrieval in agentic RAG systems. Q2D-Web tests how embedding models perform on large-scale web search usin

My XFX Radeon RX 7900 GRE 16GB Vram GPU struggles with models over 20B size. I added my Radeon RX 480 8GB Vram GPU to the system and ran a few benchmarks using llama.cpp Ubuntu Vulkan build 10453. htt

arXiv:2609.03597v1 Announce Type: new Abstract: Land ownership in Bangladesh is recorded in Ana-Ganda-Kora-Kranti-Til, a base-16 positional fraction system with dedicated Unicode glyphs, no mainstream

arXiv:2609.02339v1 Announce Type: cross Abstract: World modeling requires a predictive model to maintain and update an internal state adequate for reasoning about the consequences of actions. We intro

arXiv:2609.01982v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used as assistants for statistical and data science work, yet existing evaluations largely assume the anal

arXiv:2609.01691v1 Announce Type: new Abstract: Vision-language models (VLMs) are increasingly used to make decisions from visual inputs. We introduce FAIRLENS, a benchmark and evaluation framework fo

arXiv:2609.00048v1 Announce Type: cross Abstract: GUI world models are increasingly evaluated as one-step next-screen predictors, yet their intended use is often as multi-step environments for GUI age

arXiv:2609.00062v1 Announce Type: cross Abstract: Data contamination undermines the reliable evaluation of large language models (LLMs) on mathematical problem solving. While rewriting-based evaluatio

arXiv:2609.00330v1 Announce Type: cross Abstract: In contact centers, real-time agent-assist tools determine, for each of many predefined topics, whether a live customer utterance is relevant and disp

arXiv:2505.13180v3 Announce Type: replace Abstract: Integrating Large Language Models with symbolic planners is a promising direction for obtaining verifiable and grounded plans, with recent works ext

arXiv:2609.01056v1 Announce Type: new Abstract: Despite the growing use of LLM-powered agents to solve multi-step tasks in complex environments, existing benchmarks rarely test state preservation, per

arXiv:2608.30188v1 Announce Type: new Abstract: Large Language Models (LLMs) show great potential as clinical agents, yet existing benchmarks reduce clinical workflows to static predictions or unconst

arXiv:2608.28762v1 Announce Type: new Abstract: Recent advances in visual question answering (VQA) and multimodal large language models (MLLMs) have enabled natural-language reasoning over traffic sce

arXiv:2608.30517v1 Announce Type: new Abstract: Benchmark saturation and data contamination increasingly obscure genuine scientific reasoning in frontier LLMs. We introduce extsc{ScienceArena}, an oly

arXiv:2608.30716v1 Announce Type: new Abstract: Recent advances in Large Multimodal Models (LMMs) have greatly improved video understanding, yet their ability to reason about human-centered social sit

arXiv:2608.31108v1 Announce Type: new Abstract: Efficient evaluation changes the protocol used to support claims about model behavior, yet it is rarely tested whether those claims remain stable after

arXiv:2608.28641v1 Announce Type: cross Abstract: Most evaluations for coding agents are conducted exclusively in English, which does not reflect real-world multilingual deployment. We present Termina

arXiv:2608.29577v1 Announce Type: new Abstract: Traditional video highlight detection relies on a narrow, event-centric definition of saliency, which often fails to generalize to unconstrained persona

arXiv:2608.29411v1 Announce Type: new Abstract: Integer sequences from the On-Line Encyclopedia of Integer Sequences (OEIS) are increasingly used to benchmark mathematical reasoning in language models

The intelligence in my context is an aggregate index, I called the Agentic Coding Index, across most relevant agentic coding benchmarks: SWE-bench Pro, DeepSWE v1.1, Terminal-Bench (v4, v3, v2.1), Cod

Hey all, You’ve probably noticed that using LLMs and AI agents for pentesting has become pretty common lately. The problem is there isn’t really a good way to figure out which model is actually best s

arXiv:2608.26317v1 Announce Type: cross Abstract: Frontier language models are increasingly marketed as omni systems that can perceive and respond across modalities. Existing evaluation frameworks, ho

arXiv:2608.27394v1 Announce Type: new Abstract: Retrieved scientific literature can serve as inspiration for both human and AI scientists. Inspiration can take different forms: prior work may directly

arXiv:2608.25218v1 Announce Type: cross Abstract: Speakers in natural conversation take turns speaking and listening, deciding in real time when to take, hold, or yield the floor. However, turn-taking

arXiv:2505.11239v4 Announce Type: replace Abstract: Understanding human mobility through Point-of-Interest (POI) trajectory modeling is increasingly important for applications such as urban planning,

arXiv:2608.21060v1 Announce Type: new Abstract: Pathology foundation models (PFMs) are increasingly used as general-purpose backbones, yet existing benchmarks cannot systematically diagnose their whol

arXiv:2606.15867v2 Announce Type: replace Abstract: Multi-subject reference-based image generation requires jointly preserving multiple human identities, binding per-person objects and fashion items,

arXiv:2511.13400v3 Announce Type: replace Abstract: We introduce Embedded Stroop, a controlled diagnostic paradigm for measuring image-embedded prompt interference in Multimodal Large Language Models

https://preview.redd.it/84zi5nsdawkh1.png?width=2368&format=png&auto=webp&s=1109e69db807b153064b1f5b61d22cf1e9fbca05 Another user posted the benchmarks for Qwen 3.8 27B today, and while I think Qwen 2

New/Old benchmark that provides a lot of answers for local LLM. I present to you a new test that I developed somewhat by accident: https://huggingface.co/cHunter789/Qwen3.8-27B-i1-IQ4_KS_KT-GGUF/tree/

arXiv:2608.19957v1 Announce Type: new Abstract: Natural language code retrieval is a rapidly evolving task in computer science. However, the 1C:Enterprise ecosystem combines Russian syntax with highly

Our general-purpose coding agent just scored 100% on the ARC-AGI-3 interactive reasoning benchmark. NVIDIA AVO completed all 183 levels across all 25 public environments, figuring out what to do with

arXiv:2608.18675v1 Announce Type: new Abstract: Accurate forecasting of energy consumption is important for the efficient operation of power systems, with direct implications for operational costs, en

arXiv:2608.18423v1 Announce Type: new Abstract: Language model agents now execute bounded tasks reliably. Whether they can sustain effective decision-making over long horizons, where actions have cumu

arXiv:2608.18336v1 Announce Type: new Abstract: When evaluating language models on human exams, benchmarks typically score each response as right or wrong and report the overall accuracy. This approac

arXiv:2608.18111v1 Announce Type: new Abstract: Foundation models such as GPT and Claude now solve olympiad-level mathematics with remarkable proficiency, so much so that geometry problem solving has

arXiv:2608.16928v1 Announce Type: new Abstract: Automatic sensitivity classification of organizational documents is a critical yet underserved problem, where the consequences of misclassification rang

arXiv:2608.15326v1 Announce Type: new Abstract: Artificial intelligence (AI) benchmarks are not neutral tools of evaluation but socio-technical artefacts that shape competition, power, and research pr

arXiv:2608.14329v1 Announce Type: cross Abstract: Principle-based regulation, with evaluative standards such as 'fair, clear, and not misleading' or 'deliver good outcomes', cannot be reduced to binar