Search: “benchmark”

DRYLAB-Bench: Benchmarking the Biosecurity Risk of Agentic LLM Mutation Design

arXiv:2610.13116v1 Announce Type: new Abstract: As large language models (LLMs) acquire advanced biological design capabilities, they offer substantial research benefits but simultaneously introduce s

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

GeopoliticalQA: Benchmarking Retrieval and Generation over Intelligence Reports

DGX agent

arXiv:2610.12617v1 Announce Type: new Abstract: Geopolitical and strategic intelligence reports constitute a rich source of expert knowledge and have increasing value in the current political climate,

model-releasesarxiv-cs-cl

12 Oct 2026

Mind the Spatial Gap: A Benchmark for Geospatial Reasoning and Constrained Spatial Editing

arXiv:2610.12881v1 Announce Type: new Abstract: Understanding and manipulating spatial relations is fundamental to AI systems that reason about physical environments and construct grounded world model

Qwen 3.8 Flash Next Q3 S - Benchmark creation

model-releasesr-localllama

After running a benchmark test on Exo in Opencode and being amazed by the result, I thought I'd try other models on Opencode. Many were surprisingly terrible. Especially some mid level models. Earlier

When Charts Deceive Web Agents: A Paired Benchmark for Misleading Visualization-Grounded Tasks

arXiv:2610.12823v1 Announce Type: new Abstract: Web agents increasingly browse, interpret, and act on web content on behalf of users. In many realistic workflows, however, the visual evidence they con

Research

An Investigation of Robustness of LLMs in Mathematical Reasoning: Benchmarking with Mathematically-Equivalent Transformation of Advanced Mathematical Problems

arXiv:2508.08833v4 Announce Type: replace-cross Abstract: Frontier large language models (LLMs) now reach near-ceiling accuracy on standard mathematical-reasoning benchmarks and gold-medal-level perfo

researcharxiv-cs-ai

9 Oct 2026

Benchmark Designers Should 'Train on the Test Set' to Expose Exploitable Non-Visual Shortcuts

arXiv:2511.04655v2 Announce Type: replace Abstract: Multimodal LLMs can answer many questions in vision-centric benchmarks without looking at the image, using linguistic priors and skewed answer distr

9 Oct 2026

Alice: A Large-Scale German Benchmark for Rubric-Based Multi-Dimensional Automatic Short Answer Scoring

arXiv:2610.09661v1 Announce Type: new Abstract: Automatic Short Answer Scoring (ASAS) is central to NLP for Education. However, openly available benchmarks remain scarce, and existing datasets largely

BanglaRhet: Benchmarking Classical and Transformer Models for Rhetorical and Persuasion Detection in Bangla Political Speech

arXiv:2610.09464v1 Announce Type: new Abstract: Political discourse often uses rhetorical and persuasive language to frame narratives, influence public opinion, and mobilize audiences. While Bangla na

GPU - Vulkan llama.cpp benchmarks sorted by price to performance

model-releasesr-localllama

This table to help anyone looking to build a budget Data Center homelab. I copied the bulk of value based, mid level, decent speed results GPUs and feed it to AI or SI and here are the recommended res

Reproducible LLM Inference Benchmarking: A Sequential Isolation Protocol for Regression Testing

arXiv:2610.09778v1 Announce Type: cross Abstract: Reproducible benchmarking of Large Language Model (LLM) inference is challenging because repeated measurements can vary with execution and system stat

Benchmarking Label-Revealed Online Updates for EEG BCI Decoding

arXiv:2610.07420v1 Announce Type: new Abstract: Electroencephalography (EEG) signals drift over time, which can cause static brain-computer interface (BCI) models to degrade in practice. We present a

BiGym 2.0: Benchmarking Learned and Agent-Developed Policies for Humanoid Household Manipulation

arXiv:2610.07594v1 Announce Type: cross Abstract: Humanoid household manipulation requires the arms to act while the body balances, steps and changes posture. We present BiGym 2.0, an adaptation of Bi

How Well Do LLMs Reason with Noisy Evidence? An Active Visual Reasoning Benchmark

arXiv:2610.07751v1 Announce Type: new Abstract: Real-world reasoning rarely reduces to static question answering: agents must actively gather information from tools and sensors that are often noisy an

OTel: Open Telco AI Datasets, Benchmarks, and Models

arXiv:2610.07766v1 Announce Type: new Abstract: We present Open Telco (OTel), an open telecom AI resource that releases derived telecom datasets for retrieval, reranking, instruction tuning, and safet

The power of OpenDocRouter is already working: Anthropic just dropped Haiku 5.5 and we already have initial benchmark results. I see pretty …

model-releasesjerry-liu--x

The power of OpenDocRouter is already working: Anthropic just dropped Haiku 5.5 and we already have initial benchmark results. I see pretty comparable results to Luna overal. Slightly better at tables

ANT: A Multi-Granularity Network Traffic Dataset and Benchmark for Agents Behavior Auditing

arXiv:2610.06514v1 Announce Type: cross Abstract: The growing adoption of large language model (LLM) agents creates a need for network administrators and security teams to audit agent behavior within

Grounded in Time: A Multi-Source Dataset and Benchmark for Temporal Grounding in Robotic Manipulation

arXiv:2610.04255v1 Announce Type: cross Abstract: Robotic manipulation often requires inferring task-relevant states from past interactions when the current observation alone is insufficient to determ

Knossos and Ariadne: Benchmarking and Learning Complete Diagram Topology Extraction with Vision-Language Models

arXiv:2610.04721v1 Announce Type: new Abstract: Structural diagrams are widely used to represent complex systems and relational information across scientific, engineering, procedural, and spatial doma

MedImageOSWorld: Benchmarking GUI Agents for Medical Image Consoles

arXiv:2610.04800v1 Announce Type: cross Abstract: Graphical consoles offer a practical interface for medical acquisition assistance, allowing agents to work through the controls and visual feedback us

Beyond Single Videos: Benchmarking and Active Evidence Seeking for E-Commerce Cross-Video Reasoning

arXiv:2610.03099v1 Announce Type: cross Abstract: E-commerce videos are information-dense and frequently compared by consumers evaluating products and merchants assessing marketing strategies. However

Co-design Gym: A Unified Benchmark for Embodiment-Policy Co-optimization

arXiv:2610.02366v1 Announce Type: new Abstract: Finding an optimal behaviour policy within a given environment is a widely studied problem in domains as diverse as games, robotics, energy infrastructu

MLCommons Jailbreak Benchmark v1.0

arXiv:2610.02827v1 Announce Type: new Abstract: Modern AI systems are designed to refuse hazardous requests. A jailbreak is a prompt crafted to bypass those safeguards and elicit outputs that the syst

Threat-Preserving Representation Sensitivity in Agent-Security Benchmarks

arXiv:2610.03585v1 Announce Type: cross Abstract: Security benchmarks for LLM-based agents often report the attack success rate (ASR) as a measure of model robustness and use these scores to compare d

World Embedding Benchmark

arXiv:2610.03632v1 Announce Type: cross Abstract: Physical fidelity has received increasing attention in world models and video generation, yet how video representations encode physical information re

Build2SPARQL: A Large-Scale Text-to-SPARQL Benchmark Dataset for Building Knowledge Graph Querying

arXiv:2610.00224v1 Announce Type: new Abstract: Building automation systems are increasingly represented as semantic knowledge graphs (KGs) using ontologies such as Brick and ASHRAE 223P, creating a m

Hob-VL: A Benchmark for Visually Grounded Boolean Reasoning

arXiv:2610.01605v1 Announce Type: cross Abstract: Reliable visual reasoning requires composing multiple visual observations and returning consistent answers to logically equivalent questions. We intro

KaliBench: A Fine-Grained Benchmark for Cybersecurity Tool Use on Kali Linux with Runtime-Free Verifiable Rewards

arXiv:2610.02206v1 Announce Type: cross Abstract: LLMs are increasingly applied to cybersecurity workflows, where they are expected to translate analysts' intent into tool invocations. However, existi

MIKASA-Robo-VLA: Benchmarking Memory in VLA Models for Long-Horizon Manipulation

arXiv:2610.00604v1 Announce Type: cross Abstract: Vision-language-action policies often see only one or a few recent frames, which makes it difficult to evaluate how they use information that disappea

R-GroundBench: A Diagnostic Benchmark for R-Group Groundingin Markush Molecular Editing

arXiv:2610.00700v1 Announce Type: new Abstract: Recent advances in AI for scientific discovery enable molecular understandingand design, yet reasoning over incomplete chemical representations remainsu

RealCompanion: Benchmarking Human Understanding from Reasoning over Longitudinal Real-World Conversations

arXiv:2610.01780v1 Announce Type: new Abstract: A companion that talks with a person for months should come to understand them. It should remember what they said, infer who they are, and know when the

ChartDensity-Bench: Benchmarking MLLMs for Numerical Data Reconstruction under Visual Density

arXiv:2609.38781v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) offer a promising approach for recovering numerical data from scientific charts, but their ability to reconst

E2E-SWE: Benchmarking LLMs on Building Working Codebases from Scratch

arXiv:2609.38335v1 Announce Type: cross Abstract: Coding agents powered by large language models (LLMs) are evolving from making localized code changes to developing complete software repositories. Ho

Fiatlux: A Long-Horizon Benchmark for Humanoid Ladder Climbing and Light-Bulb Replacement

model-releasesarxiv-cs-ro

arXiv:2609.38216v1 Announce Type: new Abstract: Existing benchmarks evaluate tabletop manipulation, flat-floor household activity, or humanoid locomotion and manipulation as separate task groups; none

Front-to-Back: Benchmarking Vision-Language Models for Asymmetric Cross-View Vehicle Re-Identification

arXiv:2609.39492v1 Announce Type: new Abstract: Matching the same vehicle across front and rear cameras is difficult because the cameras do not share a view and the vehicle's appearance changes substa

Linguistic Loopholes in LLM Unlearning: From a 174-Language Benchmark to Coverage-Aware Unlearning

arXiv:2609.40286v1 Announce Type: cross Abstract: Unlearning a fact in one language does not guarantee its removal in others as changing the query or even the requested answer language can reopen seem

Sense and Sensitivity: Benchmarking LLM Clinical Triage Recommendations with Physician Experts

arXiv:2609.38600v1 Announce Type: new Abstract: As large language models (LLMs) are increasingly used in clinical settings, it is critical to evaluate their reliability under realistic variation in cl

TALK-Dem: Benchmarking Embodied Task Planning under Dementia-Associated Communication Patterns

arXiv:2609.38371v1 Announce Type: cross Abstract: Existing LLM-driven robot task planners rely on a taken-for-granted assumption of an ideal user whose instructions are clear, complete, and task-focus

ViTeX-Bench: Benchmarking High-Fidelity Video Scene Text Editing

arXiv:2609.40356v1 Announce Type: cross Abstract: Recent video generation is increasingly realistic and controllable, yet video editing remains less developed, particularly for precise local edits tha

Do Agent Benchmarks Do What They Say? An Executable-Contract Audit of Tool-Using Agent Environments

arXiv:2609.37315v1 Announce Type: cross Abstract: Tool-using agents are entering settings where a wrong action carries real cost, and the benchmarks certifying them grade what each simulated tool call

GeoOutageBench: Benchmarking Ambiguity-aware, Ontology-grounded Geospatiotemporal KGQA for Multimodal Power Outage and Resilience Analysis

arXiv:2609.36082v1 Announce Type: new Abstract: We introduce GeoOutageBench, a benchmark for assessing LLM-based geospatiotemporal KGQA for multimodal outage and resilience analysis. Unlike existing K

Google rolls out Gemini 4 Argon to a small group of cybersecurity partners and says it outperforms GPT-6 Astra on certain coding and knowledge work benchmarks (Madison Mills/Axios)

Madison Mills / Axios: Google rolls out Gemini 4 Argon to a small group of cybersecurity partners and says it outperforms GPT-6 Astra on certain coding and knowledge work benchmarks — Google is unveil

model-releasestechmeme

MCPTox: A Benchmark for Tool Poisoning Attack on Real-World MCP Servers

arXiv:2508.14925v2 Announce Type: replace-cross Abstract: By providing a standardized interface for LLM agents to interact with external tools, the Model Context Protocol (MCP) is quickly becoming a c

Sources: some Google employees say Gemini 4 performs well on benchmarks but struggles with some real-world coding tasks; Google disputes that characterization (Bloomberg)

Bloomberg: Sources: some Google employees say Gemini 4 performs well on benchmarks but struggles with some real-world coding tasks; Google disputes that characterization — As Alphabet Inc.'s Google pr

model-releasestechmeme

UserProxyBench: Evaluating LLM User Simulators for Agent Benchmarks and Training

arXiv:2609.38043v1 Announce Type: new Abstract: Interactive agent benchmarks and multi-turn reinforcement learning increasingly place a second language model in the role of the user. This simulated us

Agents

we comprehensively benchmarked gpt-6.1 sol on document OCR tasks. compared to gpt-6 sol from a week ago (!), we see a sizable increase in ta…

we comprehensively benchmarked gpt-6.1 sol on document OCR tasks. compared to gpt-6 sol from a week ago (!), we see a sizable increase in table parsing capabilities and reading order, which is pleasan

agentsjerry-liu--x

How Reusable Are Benchmarks with Richer Feedback?

arXiv:2609.32109v1 Announce Type: new Abstract: We study whether benchmarks reliably guide model selection as developers adapt to evaluation feedback across multiple criteria. We find that the worst-c

29 Sept 2026

ROTE: Benchmarking Neural Memorization on Complexity-Controlled Symbolic Sequences

arXiv:2609.31918v1 Announce Type: new Abstract: We introduce ROTE (RollOut Testing of Exact memorization), a benchmarking protocol for evaluating symbolic memorization of neural architectures. We stud