Search: “benchmark”

HaloGuard 1.0: An Open Weights Constitutional Classifier for Multilingual AI Safety

arXiv:2607.02079v1 Announce Type: new Abstract: We present HaloGuard 1.0, an open-weights implementation of the constitutional-classifier paradigm for input safety. It achieves state-of-the-art perfor

3 Jul 2026

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Research

This is the sort of early prediction you can make when you pay close attention to ARC-AGI scores

DGX agent

Francois Chollet discusses how careful observation of ARC-AGI benchmark performance can enable early predictions about AI system capabilities and progress. The post likely highlights patterns or trend

researchfrancois-chollet--x

3 Jul 2026

ALEE: Any-Language Evaluation of Embeddings via English-Centric Minimal Pairs

arXiv:2607.00171v1 Announce Type: new Abstract: Text embeddings are standard for semantic similarity tasks, yet their evaluation remains an open challenge. Current benchmarks are static, cover only a

researcharxiv-cs-cl

Beyond Document Grounding: Span-Level Hallucination Detection over Code, Tool Output, and Documents

arXiv:2607.00895v1 Announce Type: new Abstract: Hallucination detection for retrieval-augmented generation (RAG) is usually evaluated on natural-language document evidence. However, grounded generatio

Can Agents Generalize to the Open World? Unveiling the Fragility of Static Training in Tool Use

arXiv:2607.01084v1 Announce Type: new Abstract: While Large Language Model (LLM) agents demonstrate proficiency in static benchmarks, their deployment in real-world scenarios is hindered by the dynami

FusionFactory: Fusing LLM Capabilities with Multi-LLM Log Data

arXiv:2507.10540v3 Announce Type: replace Abstract: The rapid advancement of large language models (LLMs) has created a diverse landscape of models, each excelling at different tasks. This diversity d

GAIA: Geometry-Adaptive Operator Learning for Forward and Inverse Problems

arXiv:2607.01128v1 Announce Type: new Abstract: Operator learning for partial differential equations (PDEs) on arbitrary geometries builds fast neural surrogates for large-scale simulation. Although r

Interpretable vs Learned Encoders for High-Cardinality Fraud Detection

arXiv:2607.00477v1 Announce Type: new Abstract: A total of seven categorical encoding methods were tested on the IEEE-CIS fraud benchmark dataset (590,540 records, 3.5% positives, 8 high-cardinality c

Rethinking Visual Privacy: A Compositional Privacy Risk Framework for Severity Assessment with VLMs

arXiv:2603.21573v2 Announce Type: replace Abstract: Existing visual privacy benchmarks largely treat privacy as a binary property, labeling images as private or non-private based on visible sensitive

TerraBench: Can Agents Reason Over Heterogeneous Earth-System Data?

arXiv:2606.13148v2 Announce Type: replace Abstract: Climate and environmental decision-making increasingly requires reasoning across heterogeneous inputs, including gridded physical data, satellite im

Testing Frontier Large Language Models' Physics Literacy in Parallel Physical Worlds

arXiv:2607.00276v1 Announce Type: cross Abstract: Current large-language-model (LLM) physics benchmarks are usually scored by answer accuracy, which cannot distinguish genuine reasoning from recall of

Unsurprisingly, all of the strong contenders on ARC-AGI-3 so far use this type of approach.

researchfrancois-chollet--x

François Chollet observed that all leading contenders on the ARC-AGI-3 benchmark employ a similar methodological approach, suggesting a convergence in techniques for solving abstract reasoning tasks.

Tools

1/ DSGym: A Holistic Framework for Evaluating and Training Data Science Agents Paper: https://arxiv.org/abs/2601.16344

DSGym is a comprehensive framework designed to evaluate and train AI agents for data science tasks, providing a structured environment for benchmarking agent performance across various data science wo

toolstogether-ai--x

Auditing Generalization in AI-Generated Video Detection: A Six-Control Protocol and the VidAudit Toolkit

arXiv:2606.31004v1 Announce Type: new Abstract: AI-generated video detection benchmarks such as GenVidBench and AIGVDBench are the de facto leaderboards, yet most evaluation protocols leave uncontroll

Building an ASR Solution for Training and Assessing Children's Reading

arXiv:2606.31508v1 Announce Type: new Abstract: Automatic speech recognition for children's reading remains underdeveloped for most African languages, including Bambara, despite its potential value fo

CDR-Bench: Evaluating Faithful Execution of Compositional, Order-Sensitive Data Refinement Recipes

arXiv:2606.31435v1 Announce Type: new Abstract: Data refinement involves executing multi-step recipes over evolving text states, where both composition and execution order of processing operators dete

CLExEval: A Human-in-the-Loop Framework for Qualitative Evaluation of LLM Clinical Reasoning

arXiv:2606.31608v1 Announce Type: new Abstract: Large Language Models (LLMs) achieve strong results on many medical benchmarks, but their clinical reasoning remains difficult to evaluate reliably. A c

safetyarxiv-cs-cl

DetPO: In-Context Learning with Multi-Modal LLMs for Few-Shot Object Detection

arXiv:2603.23455v2 Announce Type: replace Abstract: Multi-Modal LLMs (MLLMs) demonstrate strong visual grounding capabilities on popular object detection benchmarks like OdinW-13 and RefCOCO. However,

Applications

DigitalOcean Evaluations: Production Model and Router Testing for the Inference Stack

DigitalOcean has released Evaluations, a feature for testing production models and routers within their inference stack. This tool enables developers to validate and benchmark their AI/ML models befor

applicationsdigitalocean

Modeling Cell-Cycle-Aware Single-Cell Drug Perturbation Responses

arXiv:2606.30695v1 Announce Type: cross Abstract: Single-cell drug perturbation models should predict not only transcriptional response magnitude, but also whether a treatment alters the proliferative

Multi-GPU kernels are the real test for coding models. Today at @aiDotEngineer, @simran_s_arora shared ParallelKernelBench, an open-source b…

model-releasestogether-ai--x

Multi-GPU kernels are the real test for coding models. Today at @aiDotEngineer, @simran_s_arora shared ParallelKernelBench, an open-source benchmark for evaluating whether LLMs can write fast CUDA ker

RESOLVE: A Multi-Resolution and Multi-Modal Dataset for Roadside Cooperative Perception

model-releasesarxiv-cs-cv

arXiv:2606.31895v1 Announce Type: new Abstract: LiDAR has increasingly been integrated into traffic cameras to expand coverage and mitigate occlusion in roadside cooperative perception. However, how u

Theory of Mind and Persuasion Beyond Conversation: Assessing the Capacity of LLMs to Induce Belief States via Planning and Action

arXiv:2606.31916v1 Announce Type: new Abstract: Theory of Mind (ToM) benchmarks for Large Language Models (LLMs) typically rely on passive question-answering formats, but the deployment of LLMs in inc

Xiaomi-GUI-0 Technical Report

arXiv:2606.31410v1 Announce Type: new Abstract: Graphical user interface (GUI) agents build on vision-language models to complete user tasks end-to-end in real applications through interface actions s

Can MLLMs Critique Like Humans? Evaluating Open-Ended Aesthetic Reasoning in Multimodal Large Language Models

arXiv:2606.29689v1 Announce Type: new Abstract: Open-ended aesthetic critique is a challenge for multimodal large language models (MLLMs): unlike multiple-choice aesthetic benchmarks, it has no single

safetyarxiv-cs-cl

CAN We Trust Your Results? A Cross-Dataset Study of Automotive IDS Evaluation

arXiv:2606.30430v1 Announce Type: cross Abstract: The increasing connectivity of modern vehicles has made securing in-vehicle communication networks a critical challenge. Intrusion Detection Systems (

researcharxiv-cs-lg

Clinical Reasoning Graphs: Structured Evaluation of LLM Diagnostic Reasoning Reveals Competence Without Consistency

arXiv:2606.29876v1 Announce Type: cross Abstract: Modern large language models (LLMs) reach 60-70% diagnostic accuracy on complex clinical case benchmarks, but accuracy alone cannot distinguish stable

researcharxiv-cs-ai

Data and Evaluation Closed-Loop for Model Capability Enhancement

arXiv:2606.28471v1 Announce Type: new Abstract: Model capability is the central variable in LLM pre-training, yet is never observed directly: data shapes it prospectively, while evaluation reveals it

Database Context Compression for Text-to-SQL on Real-World Large Databases

arXiv:2606.28601v1 Announce Type: cross Abstract: Recent progress in Text-to-SQL has been driven by stronger language models and prompting strategies, yet performance on real enterprise benchmarks suc

DataComp-VLM: Improved Open Datasets for Vision-Language Models

arXiv:2606.28551v1 Announce Type: cross Abstract: Building performant Vision-Language Models (VLMs) requires carefully curating large-scale training datasets, yet the community lacks systematic benchm

DIA-HARM: Dialectal Disparities in Harmful Content Detection Across 50 English Dialects

arXiv:2604.05318v2 Announce Type: replace Abstract: Harmful content detectors, particularly disinformation classifiers, are predominantly developed and evaluated on Standard American English (SAE), le

Does Verbose Chain-of-Thought Really Help? In-Distribution Evidence that Content, Not Length, Matters

arXiv:2606.30128v1 Announce Type: new Abstract: Chain-of-thought (CoT) prompting improves LLM reasoning, but the source is contested: do the intermediate steps help because they carry useful semantic

Flow Matching in Feature Space for Stochastic World Modeling

arXiv:2606.29059v1 Announce Type: cross Abstract: World modeling requires forecasting uncertain futures while preserving information useful for downstream perception. Existing visual world models ofte

Tutorials

For even higher speeds, reach out for a custom deployment! We’ve hit 446 tok/s on Artificial Analysis. Learn more → https://fireworks.ai/blo…

Fireworks AI announced achieving 446 tokens per second throughput speeds as measured by Artificial Analysis benchmarks, positioning this as their standard performance metric. The company offers custom

tutorialsfireworks-ai--x

H-GRPO: Permutation-Invariant Reinforcement Learning for Grounded Visual Reasoning

arXiv:2606.29915v1 Announce Type: new Abstract: Vision-Language Models (VLMs) often achieve high performance on benchmarks while remaining 'black boxes', yet they remain prone to hallucination or rely

Introducing GeneBench-Pro

GeneBench-Pro is a comprehensive benchmarking tool or dataset introduced by OpenAI designed to evaluate AI model performance on genomic and gene-related tasks. It likely provides standardized metrics

model-releasesopenai

Managing the Human Fallback: Skill Investment Under Improving AI and Worker Mobility

arXiv:2606.29111v1 Announce Type: new Abstract: When firms deploy autonomous AI, they must decide how much work to leave to the system and how much to keep workers engaged. This decision affects curre

NeuReasoner: Theory-grounded Mapping of Reasoning Elicitation Boundaries

arXiv:2606.29971v1 Announce Type: new Abstract: A growing body of work suggests that the reasoning capabilities of large language models are largely latent in their base form, with post-training prima

researcharxiv-cs-lg

Position: RL Researchers Need to Distinguish Between Solving Simulators and Using Simulators as a Proxy

arXiv:2606.28433v1 Announce Type: new Abstract: One goal in reinforcement learning (RL) research is to understand general-purpose sequential decision-making, using benchmark simulators as a proxy for

Reachability Guarantees for Cart-Pole Swing-Up and Stabilization

model-releasesarxiv-cs-ro

arXiv:2606.28627v1 Announce Type: cross Abstract: The cart-pole swing-up is a canonical benchmark for nonlinear control of underactuated systems, yet an end-to-end guarantee linking the global swing-u

Recursive Self-Evolving Agents via Held-Out Selection

arXiv:2606.28374v1 Announce Type: new Abstract: LLM agents are increasingly improved without weight updates by evolving a natural-language artifact, such as reflections, workflows, playbooks, cheatshe

StrucTab: A Structured Optimization Framework for Table Parsing

model-releasesarxiv-cs-cv

arXiv:2606.29905v1 Announce Type: new Abstract: Table parsing aims to convert table images into structured, machine-readable representations, a task requiring the joint perception of complex spatial l

SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads?

arXiv:2511.06090v3 Announce Type: replace-cross Abstract: Optimizing the performance of large-scale software repositories demands expertise in code reasoning and software engineering (SWE) to reduce r

TextClusterLab: An Integrated Framework for Reliable Text Clustering Studies

arXiv:2606.28328v1 Announce Type: cross Abstract: In recent years, text clustering has become a critical technique for applications including intent discovery, topic mining, and recommendation systems

Towards Physical Intuitions for Alignment Dynamics: A Case Study With Randomness Crystallization

arXiv:2606.29933v1 Announce Type: new Abstract: The alignment of language models is typically studied through the lens of capability benchmarks, but the dynamics of how models change during post-train

safetyarxiv-cs-cl

Vision-Language-Action Models: Experimental Insights from a Real-World UR5 Platform

arXiv:2606.30456v1 Announce Type: cross Abstract: This project investigates whether recent Vision-Language-Action (VLA) models can be transferred from controlled research benchmarks to a real-world ro

safetyarxiv-cs-cv

Aloe-Vision: Robust Vision-Language Models for Healthcare

arXiv:2606.27500v1 Announce Type: cross Abstract: Large Vision-Language Models (LVLMs) specialized in healthcare are emerging as a promising research direction due to their potential impact in clinica

29 Jun 2026

HumanMoveVQA: Can Video MLLMs reason about human movement in videos?