Search: “benchmark”

A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents

arXiv:2602.06052v4 Announce Type: replace-cross Abstract: Research in artificial intelligence is shifting from model innovations and benchmark scores towards problem definition and rigorous real-world

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Above-ground Biomass Estimation with Geospatial Foundation Models

DGX agent

arXiv:2608.04792v1 Announce Type: new Abstract: Accurate estimation of Above-Ground Biomass (AGB) from satellite imagery is essential for the large-scale monitoring of carbon stocks, yet it remains a

model-releasesarxiv-cs-lg

6 Aug 2026

Auto-fit vs tuned MoE offload: 564 → 1330 pp tok/s, unchanged decode (Qwen3.6-35B-A3B Q6 / RTX 3090)

model-releasesr-localllama

TL;DR: On a Qwen3.6-35B-A3B Q6 setup sized for 64K context on a 24GB RTX 3090, spilling eight MoE expert layers to CPU freed enough VRAM to increase -b from 512 to 1024 and -ub from 128 to 512. Prompt

Safety

CLIP-CC-Bench: Evaluating Paragraph-Level Video Descriptions in Video-Language Models

arXiv:2608.04302v1 Announce Type: new Abstract: Benchmarking video-language models has largely focused on short clips and single-sentence metrics, leaving open whether current systems can generate acc

safetyarxiv-cs-cv

ReGround: Restoring Visual Grounding in Multi-Step Reasoning through Self-Diagnosis and Visual Re-Examination

arXiv:2608.04385v1 Announce Type: new Abstract: Vision-Language Models (VLMs) often lose visual grounding during multi-step reasoning: as reasoning chains grow longer, later inference steps rely incre

The First EgoCross Challenge at EgoVis 2026: Cross-Domain Egocentric Video Question Answering

arXiv:2608.04589v1 Announce Type: cross Abstract: EgoCross is a cross-domain egocentric video question answering benchmark designed to evaluate whether multimodal large language models can generalize

The Yokai Learning Environment: Tracking Beliefs Over Space and Time

arXiv:2508.12480v3 Announce Type: replace Abstract: The ability to cooperate with unknown partners is a central challenge in cooperative AI and widely studied in the form of zero-shot coordination (ZS

We made an MCP for your phone. Your laptop is just half of your life, and the other half is in your phone. Now your agent gets the mobile sc…

model-releasesdiv-garg--x

We made an MCP for your phone. Your laptop is just half of your life, and the other half is in your phone. Now your agent gets the mobile screen too. No connectors, no complex setup, it has access to

Measurement Without Validity: The Compounding Reliability Problem in Agentic AI Evaluation

arXiv:2608.00794v2 Announce Type: replace Abstract: Agentic AI evaluation pipelines produce benchmark scores that justify deployment decisions, safety certifications, and regulatory compliance claims.

Applications

Modeling Scientific Experiment Scenes: Dataset and Model

arXiv:2608.02892v1 Announce Type: new Abstract: Scene Graph Generation (SGG) is fundamental to structured visual understanding, yet existing benchmarks focus mainly on daily life images and overlook s

applicationsarxiv-cs-cv

SocietyBench: Forecasting Counterfactual Social-World Evolution

arXiv:2608.04009v1 Announce Type: new Abstract: Large language models (LLMs), and the agents built on top of them, are now benchmarked heavily on whether they can finish a task -- fix a bug, drive a b

UrbanAgent: A Tool-Augmented Agent for Cross-System Urban Tasks

arXiv:2608.03018v1 Announce Type: new Abstract: Modern cities rely on an increasing number of digital services to operate, but residents' daily needs are still difficult to meet. Services are fragment

Where Did It Go Wrong? Process-Level Evaluation of Web Agents with Semantic State Tracking

arXiv:2606.15673v2 Announce Type: replace Abstract: Web agents act through long interaction sequences, yet existing benchmarks evaluate only terminal success, discarding all process information and of

WorldCup Arena: Prospective, Leakage-Free Evaluation of Frontier LLMs on a Live Tournament

arXiv:2608.04008v1 Announce Type: new Abstract: Benchmarks that measure the forecasting ability of large language models are almost always retrospective: the event has happened, the answer is somewher

A 2.6B model with tool calling and 128K context now runs at 30 tok/s on a phone

model-releasesr-localllama

Liquid AI released LFM2.5-2.6B today, and this might be more relevant to local AI than another massive model most people cannot run. The model is only 2.69B parameters, has 128K context, supports tool

Can You Trust the Confidence? ConfBench for Vision-Language Models on Document Extraction

arXiv:2608.01792v1 Announce Type: cross Abstract: Intelligent document processing (IDP) with vision-language models (VLMs) hinges on confidence scores trustworthy enough to route extractions between a

HopRefusalBench: Diagnosing Refusal Failures in Search-Augmented Agents for Multi-Hop Reasoning

arXiv:2608.01358v1 Announce Type: new Abstract: Search-augmented large language model agents are increasingly capable of solving knowledge-intensive tasks, but their behavior when a multi-hop question

MDWD: A Street-Level Dataset for Municipal Solid Waste Detection in Dense Urban Environments

arXiv:2608.00257v1 Announce Type: new Abstract: Automated visual monitoring of urban environments is a growing Computer Vision research area, but municipal solid waste detection remains under-represen

ScrambleToolBench: Agents Search Exhaustively Even When Their Own Map Points to the Next Step

arXiv:2608.02358v1 Announce Type: new Abstract: To operate robustly in open-world environments, autonomous agents should be able to infer the behavior of unfamiliar systems through interaction alone,

Safety

Semantic Alignment of AI Models: Concept Collapse, Checkpoint Dynamics, and Cross-Lingual Transfer

arXiv:2608.01585v1 Announce Type: new Abstract: Language model benchmarking is a difficult task. Outcome reasoning alone does not test the model's conceptualization of language and popular open-source

safetyarxiv-cs-cl

SpatioLM: Towards General Physical Spatial Intelligence in Vision-Language Models

arXiv:2608.01899v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) perform well on commonsense reasoning tasks but struggle with visual spatial reasoning. Most existing solutions introduc

Are the Financial Reasoning from LLMs Credible? A Real World Test over Long-Horizon Statements

arXiv:2607.28661v1 Announce Type: new Abstract: Do Large Language Models (LLMs) possess genuine structural reasoning, or merely rely on surface-level pattern matching? The financial domain, demanding

Harnessing the Wisdom of LLM Crowds through Complementarity-Driven Iterative Collaboration

arXiv:2607.29087v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed in enterprise settings, yet individual models remain bounded by model-specific capability limitat

Safety

Self-Play Meets Skill Evolution: Self-Evolving Search Agents that Pose, Solve, and Remember

arXiv:2607.29468v1 Announce Type: new Abstract: Self-play agents can generate training problems without questions from target benchmarks, but their curricula lack persistent state: failures affect gra

safetyarxiv-cs-ai

Step-Level Visual Grounding Faithfulness Predicts Out-of-Distribution Generalization in Long-Horizon Vision-Language Models

arXiv:2603.06828v2 Announce Type: replace-cross Abstract: We uncover a behavioral law of long-horizon vision-language models: models that maintain temporally grounded beliefs generalize better. Standa

WitCert: Sound Runtime Risk Observability and Gating for KV-Cache Quantization

arXiv:2607.28699v1 Announce Type: cross Abstract: KV-cache quantization is validated today by offline benchmark averages; a deployed system cannot tell whether compression is damaging the request it i

Baikal: Structured Search for Deep Research over Data Lakes

arXiv:2607.27726v1 Announce Type: cross Abstract: Deep research over data lakes requires an LLM agent to investigate evidence across thousands of heterogeneous tables and passages to synthesize a repo

Beyond Borrowed Histories: Person-Aligned User Simulation for Interactive Role-Playing Evaluation

arXiv:2607.27816v1 Announce Type: new Abstract: Role-playing agents (RPAs) have become one of the most important consumer applications of large language models. Users engage in multi-turn conversation

Beyond Similarity: Grounded Agentic Extraction and Expert-Adjudicated Evaluation of Intertextuality in Classical Chinese Histories

arXiv:2607.27595v1 Announce Type: new Abstract: Computational approaches to intertextuality have advanced from string matching to neural retrieval, yet their outputs, similarity scores and parallel-pa

CXR-Retrieve: Compositional Text-to-Image Retrieval in Chest Radiography

arXiv:2607.27779v1 Announce Type: new Abstract: Large chest radiography archives are difficult to search because most studies are paired only with free-text reports rather than structured clinical ann

DeepSeek v4 Flash has a nice bump in Capability

model-releasesr-localllama

DeepSeek V4 Flash: Preview → 2026-07-31 Benchmark Preview 0731 Δ Terminal Bench* 56.9 82.7 +25.8 Toolathlon 51.8 70.3 +18.5 NL2Repo — 54.2 new Cybergym — 76.7 new DeepSWE — 54.4 new Agent Last Exam —

DS@GT ARC at ImageCLEFmedical 2026: Architectural Diversity for Concept Detection and Foundation-Model Scaling for Caption Prediction in Medical Image Analysis

arXiv:2607.27763v1 Announce Type: new Abstract: We describe the DS@GT submissions to the ImageCLEFmedical Caption 2026 challenge, which continues a long-running benchmark on the ROCOv2 dataset with tw

EEG-EditBench: Probing Visual Information in EEG-Image Retrieval Models with Controlled Image Edits

arXiv:2607.27857v1 Announce Type: new Abstract: Recent EEG-to-image retrieval models have achieved strong performance in identifying viewed images from semantically diverse candidates. Yet such succes

Frontis-MA1: Training an AI4AI Model towards Recursive Self-Improvement in Machine Learning Engineering

arXiv:2607.28568v1 Announce Type: new Abstract: Recursive self-improvement (RSI) requires AI systems that improve the process of building AI (i.e., AI4AI); machine learning engineering (MLE) offers a

JigShape: Evaluating Visual-Geometric Reasoning in VLMs through Jigsaw Puzzles

arXiv:2607.27670v1 Announce Type: new Abstract: Jigsaw puzzle solving requires jointly reasoning about visual content and geometric constraints, yet existing benchmarks use rectangular cuts that creat

What’s new in AI infrastructure and orchestration this month

model-releasesgoogle-cloud-ai

At Google, AI is a soup-to-nuts endeavor. Obviously, we make leading AI models like Gemini and Nano Banana. We incorporate AI into the tools you use every day (think Gmail, BigQuery, AlloyDB, Google C

Diagnosing Fine-Grained Inconsistency Classification in Financial Disclosure Text

arXiv:2607.26368v1 Announce Type: new Abstract: Financial disclosures contain numerical claims, temporal statements, entity references, policy commitments, and risk descriptions that may conflict in q

MediaWiki Code2Code Search: Neural Retrieval for the Semantic Discovery of Open-Source Software Entities

arXiv:2607.26766v1 Announce Type: cross Abstract: Code search in large-scale ecosystems is often hindered by the lexical gap between user queries and implementation details, alongside the trade-off be

SERPO: Self-Evolving Rubric Policy Optimization for Open-Ended Test-Time Reinforcement Learning

arXiv:2607.26873v1 Announce Type: new Abstract: Test-time reinforcement learning (TTRL) enables language models to self-evolve at inference time without labeled feedback. Existing methods rely on answ

Thinking Machines just released Inkling-Small: 276B total, 12B active. A faster Inkling that matches or beats its 975B big brother in many b…

model-releasessoumith-chintala--x

Thinking Machines just released Inkling-Small: 276B total, 12B active. A faster Inkling that matches or beats its 975B big brother in many benchmarks. To test its speed, we plugged it into HF's speech

Agentic AI for Scientific Reasoning in Autonomous Quantum Sensing Experiments

arXiv:2607.25145v1 Announce Type: cross Abstract: We implement an agentic AI workflow built around a large language model (LLM) agent for autonomous experiments with nitrogen-vacancy (NV) centers in d

29 Jul 2026

CARE-MH: Towards Unified, Reproducible, and Comparable Evaluation of Mental Health LLMs

arXiv:2607.24754v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used to provide mental health support, requiring reliable evaluation of safety, empathy, and therapeutic

29 Jul 2026

Conformal Cascade: Distribution-Free Accuracy Guarantees for Multi-Tier LLM Inference

model-releasesarxiv-cs-lg

arXiv:2607.25018v1 Announce Type: new Abstract: Large language model (LLM) cascades reduce inference cost by routing easy queries to a small model and deferring hard queries to a larger one. Productio

29 Jul 2026

An Ontology for Machine Learning Interatomic Potentials

arXiv:2607.23219v1 Announce Type: new Abstract: Machine learning interatomic potentials (MLIPs) approximate quantum-mechanical energies and forces---conventionally computed by density functional theor

AptAvatar: Fast and Vivid Long-Form Audio-Driven Video Generation for Production-Ready Avatars

arXiv:2607.24013v1 Announce Type: new Abstract: Production-ready audio-driven avatar generation requires efficient inference without sacrificing fidelity or motion expressiveness. However, existing ac

AssumptionMiner: Extracting, Tracing, and Revising Implicit Assumptions in LLM Code Generation

arXiv:2607.22898v1 Announce Type: cross Abstract: Large language models (LLMs) generate code from natural-language prompts, yet real-world prompts rarely provide complete specifications. When prompts

DOSA: A Tree-Guided, Self-Regressive Framework for Long Document Structure Analysis

arXiv:2607.22679v1 Announce Type: new Abstract: In visually-rich documents, information is encoded not only in individual page objects such as tables, headers, and text blocks, but also in the structu

Hierarchical Group-Conditional Conformal Risk Control for Selective Prediction in Language Models

arXiv:2607.24562v1 Announce Type: new Abstract: Large language models serve heterogeneous populations structured by domain, topic difficulty, and linguistic style. Conformal risk control (CRC) gives r