Search: “benchmark”

JEV almost dead: CLM vs JEV

Original post: https://www.reddit.com/r/LocalLLaMA/comments/1woscea/contrastive_language_models/ (sorry I felt it wasn't giving CLM the highlight it deserves) What it is: a new projection head for Qwe

Live Assistant: Learning Whether, When, and Whom to Assist in Real-World Live Social Streams

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

DGX agent

arXiv:2609.27303v1 Announce Type: new Abstract: Livestreams are long-lasting interactive environments where audiovisual content, viewer activity, host behavior, and platform signals evolve together, c

model-releasesarxiv-cs-lg

24 Sept 2026

MolDesignBench: Evaluating LLM-based Agent for Scenario-grounded Molecular Design

arXiv:2609.27349v1 Announce Type: new Abstract: Real-world molecular design remains challenging for large language model (LLM)-based agents. It requires them to interpret design contexts, satisfy mult

Pro-Bench: Prompt-Robust Open-Vocabulary Visual Grounding Across Real-World Heterogeneous Environments

arXiv:2609.27076v1 Announce Type: new Abstract: Open-vocabulary visual grounding enables robots to localise task-relevant entities from natural-language queries without dependence on predefined percep

Provably Complete Generalized Planning with LLMs

arXiv:2609.27105v1 Announce Type: new Abstract: Generalized planning aims to compute a plan that solves all instances of a planning domain. Recent work has used LLMs to automatically generate and debu

Schrodinger's Code Repository: Have LLMs Learned SWE-bench or Memorized It?

arXiv:2609.27891v1 Announce Type: cross Abstract: Repository-level coding benchmarks have become the standard for evaluating coding agents, yet they inherently suffer from data leakage because they ar

Silent Failures in Agent-Tool Interaction: An Audit of ToolUniverse

arXiv:2609.26836v1 Announce Type: new Abstract: Agentic AI systems are increasingly adopting automated pipelines that integrate multiple tools. While prior research and benchmarks have studied about t

Spec2COBOLRot: An Agentic-AI Degradation Loop for Realistic COBOL Corpus Generation

arXiv:2609.26835v1 Announce Type: cross Abstract: COBOL remains widely deployed, yet representative corpora reflecting real production code are rarely available, limiting rigorous benchmarking of mode

UkisAI Swift Series / 27B, Flash Next and Bonsai 2 + GSQ-RCO / -63.4% thinking, x1.95 speed with xhigh accuracy

Hey everyone, Jovan from UkisAI here! Today, we are introducing Swift, a family of efficient reasoning LLMs based on Qwen, trained by penalizing tokens related to pathological overthinking patterns an

Visual Tripwires: Anticipating Failure in Deep Vision Systems

arXiv:2609.28099v1 Announce Type: new Abstract: Deep vision systems remain vulnerable to corruption, occlusion, and distribution shift despite strong benchmark performance. Existing reliability method

VLM2GeoVec: Toward Universal Multimodal Embeddings for Remote Sensing

arXiv:2512.11490v2 Announce Type: replace Abstract: Satellite imagery differs from natural images in viewpoint, resolution, scale variation, and the prevalence of small objects -- demanding both regio

When Labels Are Scarce: An Oscillatory State Space Model for Vibration Diagnosis

arXiv:2609.27411v1 Announce Type: new Abstract: Machine fault diagnosis from vibration requires learning from scarce labelled fault recordings while meeting the computational constraints of edge devic

Beyond Static Charts: Can Language and Vision Language Models Generate Interactive Data Visualization Interfaces?

arXiv:2609.26208v1 Announce Type: new Abstract: Data visualization is central to analytical reasoning, but real-world analysis increasingly requires language-driven interactive interfaces rather than

Certified Against Which Oracle? Execution Labels Set the Reported Risk of Conformal Abstention for Text-to-SQL

arXiv:2609.25938v1 Announce Type: cross Abstract: A conformal abstention certificate for text-to-SQL is only as truthful as the correctness labels it is calibrated on. The uncertainty pipelines that r

Deep Reinforcement Learning on Item-Compatibility Graphs for One-Dimensional Bin Packing

arXiv:2609.25397v1 Announce Type: new Abstract: The one-dimensional bin packing problem (1D-BPP) is a classical NP-hard combinatorial optimization problem with applications ranging from logistics and

FrontierMath Erdos

arXiv:2609.25050v1 Announce Type: new Abstract: We introduce FrontierMath Erdos (FME), a benchmark of 68 Erdos problems that are open as of August 2026. To solve a task in FME, AI systems must resolve

MiMo-V2.6 (both Pro and Flash) is a benchmaxxed scam

MiMo-V2.6-Pro has an insanely high score of 46 on AA, putting it at the head of the opensource models available. It also costs pennies. Flash is not out on AA yet, but it costs less than half on datac

MiMo-V2.6-Pro is live on Wally. Day 1. @XiaomiMiMo's new open-weights flagship, top of the @ArtificialAnlys Intelligence Index for open-weig…

model-releasesyohei-nakajima--x

MiMo-V2.6-Pro is live on Wally. Day 1. @XiaomiMiMo's new open-weights flagship, top of the @ArtificialAnlys Intelligence Index for open-weight models. 320 tok/s, 2.4x the model maker's own API, in <24

Modality-Gated Deep Adapters: Adding a Modality to a Frozen Embedding Model with Exact Preservation

arXiv:2609.26182v1 Announce Type: new Abstract: Multimodal embedding models are deployed at scale: retrieval indices, benchmark results, and behavioral audits all depend on the base model's exact outp

Must-read paper from Google on self-improving agent harnesses. If you auto-optimize your agent's harness, your eval score can go up while th…

Must-read paper from Google on self-improving agent harnesses. If you auto-optimize your agent's harness, your eval score can go up while the agent gets worse on real tasks. This paper shows how to pr

model-releasesdair-ai--x

Real-World Perception for Autonomous Driving in Adverse Weather: Enhancing Standard Detectors via Foundation-Guided Auto-Annotation

arXiv:2609.25515v1 Announce Type: new Abstract: Standard deployment-ready object detectors for autonomous vehicles degrade in adverse weather and lighting conditions without being trained on extensive

The Endless Exam: Mathematical Constructions from Today's Models toward Superintelligence

arXiv:2609.24555v1 Announce Type: new Abstract: We introduce the Endless Exam, a benchmark for measuring mathematical progress from today's models toward artificial superintelligence through fourteen

Variational objectives for amortized Bayesian inference in inverse problems: The role of posterior conditioning

arXiv:2609.25145v1 Announce Type: cross Abstract: Variational autoencoders (VAEs) offer an efficient approach to amortized Bayesian inference for inverse problems, but posterior accuracy can depend st

We’re demonstrating how frontier models have continued to improve in realistic mental health conversations with MentalHealthBench. This new …

We’re demonstrating how frontier models have continued to improve in realistic mental health conversations with MentalHealthBench. This new open benchmark was built with input from more than 80 mental

model-releasesopenai--x

WorkWorlds: An Infrastructure for Evaluating AI Agents on Workplace Tasks

arXiv:2609.23806v1 Announce Type: new Abstract: Many knowledge-work benchmarks are constructed around individual tasks, with the context needed for each task selected together with or after the task h

Anthropic's Claude Opus 5.5 is now available in Hermes Agent via Nous Portal!

model-releasesnous-research--x

Anthropic's Claude Opus 5.5 is now available in Hermes Agent via Nous Portal! OPUS 5.5 BENCHMARKS JUST DROPPED. It beats Fable 5.1 and GPT 6 Astra across the board. Agentic coding, knowledge work, rea

CE^4L: Continual Ego, Exo, and Ego-Exo Learning

arXiv:2609.23492v1 Announce Type: new Abstract: Perception for embodied agents is video-based, often multi-view (ego, exo, or both), and inherently continual, with simultaneous task and viewpoint shif

Dynamic Quantiser - a way to make your own high quality dynamic quants

Dynamic Quantiser TLDR - Makes dynamic and/or custom quants of ggufs. Doesn't need data, just gguf file + llama.cpp. Minimises cosine deviation - highly correlated with minimising KLD. Fast, much bett

Safety

GRACE: Grounded Adversarial Reasoning over Canadian Law

arXiv:2609.23726v1 Announce Type: new Abstract: Large language models have shown strong performance across a range of legal tasks, but existing benchmarks rarely evaluate the ability to take and defen

safetyarxiv-cs-cl

IMPLICIT-Bench: Measuring Implicit Bias in Text-to-Image Models under Neutral Prompts

arXiv:2609.24228v1 Announce Type: new Abstract: Text-to-image (T2I) models are typically evaluated for bias using slot-based templates such as ``a photo of a [profession]''. Such templates probe only

Introducing DigitalOcean Managed Agents: One AI-native stack to power your intelligence

model-releasesdigitalocean

Following a successful private preview, we’re thrilled to open DigitalOcean Managed Agents to everyone. Teams can now deploy their preferred agent harness (like OpenCode, Codex CLI) or bring their own

Is Imagination Derived from Hallucination? A Cross-Taxonomy Evaluation of Imagination and Hallucination in Large Language Models

arXiv:2609.22152v1 Announce Type: new Abstract: Imagination performs as a high-level function of large language models (LLMs) which determines the potential of how an LLM creates unseen or creative co

Learning and Control Beyond Linearity: Towards a Non-asymptotic Theory for Bilinear Systems

arXiv:2609.22338v1 Announce Type: cross Abstract: This tutorial provides a unified view of the emerging area of bilinear learning and control. Using linear systems as a benchmark, it explains what fun

Research

Matched-Input Estimates Differ in Sign Across Architectures: Auditing EEG Foundation Models on Motor Imagery

arXiv:2609.23924v1 Announce Type: new Abstract: Pretrained EEG foundation models are increasingly proposed as general-purpose encoders for brain-computer interfaces, yet recent benchmarks disagree abo

researcharxiv-cs-lg

Monitorable Chart Reasoning Agents via Verifiable Process Rewards

arXiv:2609.24071v1 Announce Type: new Abstract: Chart reasoning agents are increasingly used to extract actionable insights in critical domains, achieving state-of-the-art performance on multiple benc

Multicentre Bi-atrial Segmentation from LGE-MRI for Atrial Fibrillation with a 2D and 3D Framework

arXiv:2609.22398v1 Announce Type: cross Abstract: Accurate delineation of bi-atrial structures from late gadolinium enhancement MRI (LGE-MRI) is an important prerequisite for structural analysis and f

Safety

ReSTI: A Source-Grounded Audit and Repair of STI-Bench

arXiv:2609.24727v1 Announce Type: new Abstract: Spatial--temporal benchmarks are valid only when their questions, source annotations, and answer options identify the same physical quantity. We audit S

safetyarxiv-cs-cv

Industry

Right-size generative AI endpoints with concurrency sweeps on Amazon SageMaker AI

Concurrency sweeps help you right-size a generative AI endpoint on Amazon SageMaker AI by systematically benchmarking it at increasing load levels. This post walks through deploying a model, running a

industryaws-ml-blog

XYEval: Agents say yes to bad advice

arXiv:2609.23939v1 Announce Type: new Abstract: Effective communication between users and AI agents is essential for human-AI collaboration. The XY problem is a well-known communication pitfall where

A Scene Language Model for Open-Vocabulary Scene Mapping

arXiv:2609.21400v1 Announce Type: new Abstract: Open-vocabulary 3D scene mapping aims to build a persistent representation of the objects in an environment. Existing systems typically rely on engineer

EnSol: an environment-aware graph neural network for molecular solubility prediction

arXiv:2609.21151v1 Announce Type: cross Abstract: Molecular solubility directly affects key aspects of molecular development such as reaction feasibility, formulation performance, separation efficienc

Global AI routing with <1% overhead on multi-cluster GKE Inference Gateway

model-releasesgoogle-cloud-ai

Demand for AI infrastructure is at an all-time high. Global accelerator shortages mean engineering teams can rarely get all the compute they need from just one data center — capacity comes a cluster h

Impressive paper on building recursive self-improving agent harnesses. It's rich with great insights on building effective agent harnesses. …

Impressive paper on building recursive self-improving agent harnesses. It's rich with great insights on building effective agent harnesses. If you maintain an agent harness, this paper names three def

model-releasesdair-ai--x

Not All Irregularity Is Equal: Causally Isolating a Rare Failure Mode in Japanese Morphological Inflection

arXiv:2609.21179v1 Announce Type: new Abstract: Neural morphological generation systems often achieve high aggregate accuracy on benchmark datasets, yet such performance can conceal systematic errors

Scale your AI workloads faster and more efficiently with GKE Pod snapshots

model-releasesgoogle-cloud-ai

When running modern AI workloads, there’s often a conflict between performance and cost. Workloads like large language models (LLMs) load massive files, and may serve thousands of AI agents that need

SWE-Proof: Can Language Models Resolve Real-World Issues with Machine-Checked Proofs?

arXiv:2609.21190v1 Announce Type: cross Abstract: Ensuring the correctness of LLM-generated code is a core challenge for modern software engineering. Benchmarks for agentic code generation check corre

VISPATH: Visual-Intent-Guided Path Reasoning for Multimodal Knowledge Graph Question Answering

arXiv:2609.20843v1 Announce Type: new Abstract: Knowledge graph question answering (KGQA) enables models to answer natural-language questions through structured graph reasoning and has achieved substa

DIY Jev

So this jev thingy is getting kind of big... tbh it seems overhyped by a large margin, but here we are. Not that its bad, just feels like we usually ignored larger things... Anyway to the point of thi