Search: “benchmark”

Fair on the Surface? Benchmarking Hidden-Output Fairness Gaps in LLM Recommenders

arXiv:2608.08284v1 Announce Type: new Abstract: Fairness audits for LLM-based recommenders have largely focused on observable outputs, implicitly assuming that stable recommendations reflect stable in

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

From Values to Benchmarks: Evaluating Large Language Models for Governmental Use in Dutch

DGX agent

arXiv:2608.09925v1 Announce Type: cross Abstract: Large language models are increasingly being deployed in governmental settings, yet few existing evaluation frameworks jointly reflect the values of p

model-releasesarxiv-cs-ai

11 Aug 2026

Gradient Under Microscope: Benchmarking Resource Utilization of Memory-Efficient Gradient Computation Methods

arXiv:2608.08961v1 Announce Type: new Abstract: AI training's rising resource intensity is straining electricity supplies and carbon budgets, motivating systematic study of memory-efficient training o

KVDiagnosis: A Diagnostic Benchmark for KV-Cache Compression in Long-Context Language Models

arXiv:2608.09412v1 Announce Type: new Abstract: KV-cache compression reduces long-context memory, but aggregate task scores reveal neither which correct executions fail nor why. We present KVDiagnosis

Mawqif-v2: An Arabic Benchmark Dataset for Cross-Target Stance Detection

arXiv:2608.09539v1 Announce Type: new Abstract: Publicly available Arabic datasets for target-specific stance detection remain limited, particularly for evaluating cross-target generalization. This pa

Rethinking 3D Segmentation from Individual LiDAR Scans: Incidence-Aware Sampling on the SIP Benchmark

arXiv:2608.07757v1 Announce Type: new Abstract: 3D scene understanding is increasingly important in construction, yet most methods are developed on curated datasets that do not fully reflect real site

SafeSceneReason: A Multimodal Reasoning Benchmark Connecting Industrial Hazards with Accident Knowledge

arXiv:2608.09230v1 Announce Type: new Abstract: Industrial-safety understanding requires more than detecting workers, equipment, and personal protective equipment. Models must also assess compliance,

Social Gym and SPaRTan: Benchmarking and Improving LLM Social Reasoning via Multi-Agent Game Tournaments

arXiv:2608.09128v1 Announce Type: cross Abstract: LLM agents are increasingly deployed in multi-agent social settings where they must cooperate, negotiate, and adapt to other agents. Measuring and imp

Temporal Generalization in fNIRS-Based Autism Classification: A Cross-Time-Window Transfer Benchmark

arXiv:2608.07567v1 Announce Type: cross Abstract: Functional near-infrared spectroscopy (fNIRS) is a promising modality for autism spectrum disorder (ASD) classification, yet existing approaches assum

Temporal Misgrounding in Legal RAG: A Versioned-Corpus Benchmark for French Tax Law

arXiv:2608.09393v1 Announce Type: cross Abstract: We identify and quantify temporal misgrounding: the systematic retrieval and citation of the currently in-force version of a legal article when the ap

Time Present and Time Past: Benchmarking Large Language Models on Temporally Evolving Document Understanding

arXiv:2608.08512v1 Announce Type: new Abstract: Evolving documents, such as laws, tax codes, and software documentation, are amended, replaced, and sometimes reverted over time, so a question has diff

VideoVIBE: A Video-Grounded Diagnostic Benchmark for One-Shot Interactive Website Generation

arXiv:2608.09573v1 Announce Type: new Abstract: Natural-language-driven 'vibe coding' enables the one-shot generation of visually rich and interactive web applications, yet reliable assessment of thei

When Skills Meet Safety: Benchmarking and Characterizing the Adaptive Jailbreak Robustness of Skill-Merged LLMs

arXiv:2608.08542v1 Announce Type: new Abstract: Model merging has become the default way to give an aligned language model new skills without retraining: a practitioner folds task vectors from math, c

Beyond Fluency: A Clinical Benchmark and Anomaly-Enhanced Baseline for Spine MRI Report Generation

arXiv:2608.07117v1 Announce Type: new Abstract: Radiology reporting is time-consuming and subject to inter-rater variability, making automated report generation an attractive clinical application for

10 Aug 2026

Do AI Personas Grow? Analyzing and Benchmarking Personality Evolution in LLM Agents After Life Events

arXiv:2608.06485v1 Announce Type: cross Abstract: Personality-conditioned LLM agents (PC-Agents) are increasingly used in emotional support, social simulation, and role-playing, motivating the develop

10 Aug 2026

LitTraceQA: A Benchmark for Multi-Stage Grounding and Verification in Scientific Question Answering

arXiv:2608.07370v1 Announce Type: new Abstract: Scientific literature is increasingly used as a knowledge source for language models, retrieval-augmented generation systems, and research assistants, b

10 Aug 2026

Big, Bright, or Invisible: A Frozen-Feature Benchmark of 3D CT Foundation Models

arXiv:2608.05960v1 Announce Type: cross Abstract: Routine CT interpretation is inherently comprehensive, capturing incidental findings across the entire scan volume. 3D CT foundation models could assi

Enhancing Anomaly Resilience in Research Networks: A Large-Scale Forecasting Benchmark for Dynamic Security Baselining

arXiv:2608.05605v1 Announce Type: cross Abstract: Research and Education Networks (RENs) serve as critical infrastructure for scientific discovery, yet they face a unique security paradox: their norma

Evaluating Investment Logic in Large Language Models: A Real-World Benchmark Towards Personalzied Financial Agents

arXiv:2608.06108v1 Announce Type: new Abstract: Investment competence is inherently personalized: the same market evidence can justify different actions for investors with different goals, horizons, p

From Sports to Safety: Benchmarking Proactive Risk Inference in MLLMs

arXiv:2608.05560v1 Announce Type: cross Abstract: Timely anticipation of physical hazards is essential for real-world safety, yet existing MLLM evaluations focus on harmful content or general risks, l

LabyrinthBench: a local-focused, judge-free LLM benchmark that measures context recall under interference for multi-step agentic tasks.

model-releasesr-localllama

LabyrinthBench measures the thing that actually kills long agent runs — whether a model can still use what it learned twenty turns ago — deterministically, with no LLM judge, on your own hardware, wit

PoolBench: A Benchmark for Pooling Strategies in Concept Representation Evaluation for Decoder-Only LLMs

arXiv:2608.05162v1 Announce Type: new Abstract: Pooling is a consequential but under-examined design choice in decoder-only concept representation work: practitioners must collapse token-level hidden

Who Checks the Citations? Benchmarking Legal Hallucination Detection

arXiv:2606.21155v2 Announce Type: replace Abstract: Attorneys, judges, and pro se filers increasingly use AI to draft legal documents, yet these tools frequently fabricate citations. Despite predictio

ContextWeave: A Real-World Workflow Benchmark

arXiv:2608.04830v1 Announce Type: new Abstract: Memory is essential as language agents move from isolated tasks to long-horizon, stateful workflows, yet existing evaluations often reduce it to retriev

Research

DeepAmbigQA: Ambiguous Multi-hop Questions for Benchmarking LLM Answer Completeness

researchapple-ml-research

Large language models (LLMs) with integrated search tools show strong promise in open-domain question answering (QA), yet they often struggle to produce complete answer set to complex questions such a

Easy to Complete, Hard to Choose: Investigating LLM Performance on the ProverbIT Benchmark

arXiv:2608.04670v1 Announce Type: cross Abstract: Large Language Models (LLMs) have transformed computational linguistics and achieved remarkable performance across numerous natural language processin

From Financial Sentiment Classification to Return Predictability: A QLoRA Benchmark of Large Language Models

arXiv:2608.04200v1 Announce Type: cross Abstract: Financial sentiment classifiers are commonly evaluated against human labels, but strong linguistic performance does not necessarily imply economically

Towards a satellite image manipulation and deepfake localization benchmark dataset

arXiv:2608.04840v1 Announce Type: cross Abstract: Verifying the authenticity of satellite imagery has become increasingly critical given advances in generative artificial intelligence. Highly realisti

Two-Level Meta-Rubrics for Evaluating Open-Ended Generation: GAMUT, a Benchmark for Factual Completeness

arXiv:2607.19322v2 Announce Type: replace Abstract: Rubric-based evaluation of open-ended generation faces a fundamental tension between expressiveness and reliability. Authoring a faithful rubric req

Agents Catching Agents: Shortcut Cascades and Benchmark Gaming in Clinical Multi-Agent Systems

arXiv:2608.03744v1 Announce Type: new Abstract: Clinical decision support is moving toward committees of language-model agents deliberating on a shared workspace. We ask whether such committees can be

BanglaWild: An In-the-Wild Bengali Scene Text Recognition Benchmark for OCR and Vision-Language Models

arXiv:2608.03884v1 Announce Type: cross Abstract: In-the-wild Bengali scene text recognition is largely unmeasured: existing resources target handwritten documents or constrained sign-board parsing, r

CARE-Bench: Benchmarking Patient-Facing LLM Triage

arXiv:2608.03731v1 Announce Type: new Abstract: Patient-facing medical LLMs and agents increasingly answer symptom questions before clinician contact, where the key safety question is what action the

Research

FOUND-AF: Benchmarking ECG Foundation Models for Atrial Fibrillation Detection

arXiv:2608.03597v1 Announce Type: new Abstract: Atrial fibrillation (AF) is the most common sustained cardiac arrhythmia and is associated with increased risks of stroke, heart failure, and mortality.

researcharxiv-cs-ai

PAST-Bench: Benchmarking the Foundations of Recursive Self-Improvement in Personal Agents

arXiv:2608.04003v1 Announce Type: new Abstract: Recursive self-improvement requires agents to turn accumulated experience into better future behavior. Personal AI agents offer a concrete setting for s

Security-First Evaluation of Text-to-Terraform: Benchmarking LLMs and SLMs for Secure IaC Generation

arXiv:2608.02672v1 Announce Type: cross Abstract: Cloud misconfiguration remains a leading cause of security incidents, yet whether LLMs and SLMs can generate security-compliant Infrastructure-as-Code

When Classes Evolve: A Benchmark and Framework for Stage-Aware Class-Incremental Learning

arXiv:2602.00573v2 Announce Type: replace-cross Abstract: Class-Incremental Learning (CIL) aims to sequentially learn new classes while mitigating catastrophic forgetting of previously learned knowled

When Memory Becomes Authority: Benchmarking Authority Collapse at the Memory Consolidation Boundary

arXiv:2608.01679v2 Announce Type: replace Abstract: Persistent memory allows (self-evolving) LLM agents to adapt across tasks by consolidating heterogeneous interaction histories into reusable facts,

Benchmarking Sheaf Neural Networks for Inductive Tasks

arXiv:2608.02558v1 Announce Type: new Abstract: Sheaf Neural Networks (SNNs) generalize message passing by replacing scalar edge weights of standard Graph Neural Networks (GNNs) with learnable, edge-d

From Global to Local: A Scalable Benchmark for Local Posterior Sampling

arXiv:2507.21449v2 Announce Type: replace-cross Abstract: Degeneracy is an inherent feature of the loss landscape of neural networks, but it is not well understood how stochastic gradient MCMC (SGMCMC

Lethe: How Hard Is It to Forget? A Benchmark for Federated Unlearning in Medical Imaging

arXiv:2608.01094v1 Announce Type: new Abstract: Federated learning enables medical-imaging models to be trained across hospitals, and privacy law, most explicitly the GDPR ``right to be forgotten'', t

onepot-Bench 0: towards lab-aware in silico chemistry benchmarks

arXiv:2608.02595v1 Announce Type: new Abstract: Language models are playing an increasingly important role in laboratory science, performing tasks such as experiment planning, execution, and post-hoc

SVGEval: A Vision-Grounded Framework for Perceptual-Quality Benchmarking and Evaluation in Text-to-SVG Generation

arXiv:2608.01977v1 Announce Type: new Abstract: Multimodal large models are increasingly used to generate scalable vector graphics (SVG), but reliable evaluation remains underexplored. Existing protoc

A Unified Benchmark of Deep Learning Models for Multi-task 3D Brain Tumor Segmentation from Magnetic Resonance Imaging

arXiv:2607.28858v1 Announce Type: cross Abstract: Automatic brain tumor segmentation from magnetic resonance imaging (MRI) has become a fundamental task in computer-assisted diagnosis, treatment plann

'Data center in a Box (on Wheels)' 256Gb VRAM/512Gb RAM AI Server 6-8 Month Operational Review, Stability Write Up, Benchmarks

model-releasesr-localllama

I've been out of these forums for awhile but I figured I would provide a formal update on how this has been going now that it has some operation time under its belt, just to put the information out th

Hardware

NVIDIA Vera Storage Benchmarks: Faster Encryption, Compression, Integrity Checking, and Recovery for AI-Native Storage

NVIDIA’s Vera BlueField‑4 STX Storage Processor combines an 88‑core Armv9.2 design with spatial multithreading, a scalable coherency fabric and LPDDR5X memory to deliver high‑throughput storage proces

hardwarenvidia-developer

Simulation Code Generation for Fluid Systems using Large Language Models: Benchmarking Models and Prompting Strategies

arXiv:2607.29389v1 Announce Type: new Abstract: Large language models (LLMs) have demonstrated a strong ability to generate syntactically correct code from natural-language specifications. In this stu

Research

TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs

arXiv:2607.28640v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) should generate consistent responses given semantically equivalent inputs across modalities. However, we observ

researcharxiv-cs-cl

Benchmarking Foundation and Large Language Models for Few-Shot Medical Image Segmentation

arXiv:2607.27856v1 Announce Type: new Abstract: Few-shot medical image segmentation (FS-MIS) aims to segment novel regions of interest (ROIs) from a few annotated support examples. Despite rapid progr