Search: “benchmark”

Toward Culturally Grounded Natural Language Processing

arXiv:2603.26013v2 Announce Type: replace Abstract: Multilingual NLP is often treated as a route to global inclusion, but linguistic coverage and cultural competence frequently diverge. This paper syn

5 May 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Evaluating the Architectural Reasoning Capabilities of LLM Provers via the Obfuscated Natural Number Game

DGX agent

arXiv:2605.00677v1 Announce Type: new Abstract: While Large Language Models have achieved notable success on formal mathematics benchmarks such as MiniF2F, it remains unclear whether these results ste

model-releasesarxiv-cs-lg

4 May 2026

ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat Investigation

arXiv:2507.14201v3 Announce Type: replace-cross Abstract: We present ExCyTIn-Bench, the first benchmark to Evaluate an LLM agent X on the task of Cyber Threat Investigation through security questions

4 May 2026

SCAN: Structured Capability Assessment and Navigation for LLMs

arXiv:2505.06698v4 Announce Type: replace Abstract: Evaluating Large Language Models (LLMs) has become increasingly important, with automatic evaluation benchmarks gaining prominence as alternatives t

researcharxiv-cs-cl

4 May 2026

Try Grok

model-releaseselon-musk--x

Try Grok Grok 4.3 just became the smartest AI in the world at law and money It took #1 on TWO brutal private tests no other model could win on “Vals AI” benchmarks #1 CaseLaw (v2) - 79.31% accuracy Pr

4 May 2026

Activation Function Design Sustains Plasticity in Continual Learning

arXiv:2509.22562v4 Announce Type: replace-cross Abstract: In independent, identically distributed (i.i.d.) training regimes, activation functions have been benchmarked extensively, and their differenc

researcharxiv-cs-ai

1 May 2026

AQUA-Bench: Beyond Finding Answers to Knowing When There Are None in Audio Question Answering

arXiv:2601.12248v2 Announce Type: replace-cross Abstract: Recent advances in audio-aware large language models have shown strong performance on audio question answering. However, existing benchmarks m

Evaluating LLM Safety Under Repeated Inference via Accelerated Prompt Stress Testing

model-releasesarxiv-cs-lg

arXiv:2602.11786v2 Announce Type: replace Abstract: Traditional benchmarks for large language models (LLMs), such as HELM and AIR-BENCH, primarily assess safety through breadth-oriented evaluation acr

Frontier Coding Agents Can Now Implement an AlphaZero Self-Play Machine Learning Pipeline For Connect Four That Performs Comparably to an External Solver

model-releasesarxiv-cs-lg

arXiv:2604.25067v1 Announce Type: cross Abstract: Forecasting when AI systems will become capable of meaningfully accelerating AI research is a central challenge for AI safety. Existing benchmarks mea

Walking Through Uncertainty: An Empirical Study of Uncertainty Estimation for Audio-Aware Large Language Models

arXiv:2604.25591v1 Announce Type: cross Abstract: Recent audio-aware large language models (ALLMs) have demonstrated strong capabilities across diverse audio understanding and reasoning tasks, but the

A Large-Scale, Cross-Disciplinary Corpus of Systematic Reviews

arXiv:2604.22864v1 Announce Type: cross Abstract: Existing benchmarks for systematic reviewing remain limited either in scale or in disciplinary coverage, with some collections comprising only a modes

researcharxiv-cs-cl

Detecting and Evaluating Medical Hallucinations in Large Vision Language Models

model-releasesarxiv-cs-cv

arXiv:2406.10185v2 Announce Type: replace Abstract: Large Vision Language Models (LVLMs) are increasingly integral to healthcare applications, including medical visual question answering and imaging r

Let's talk document formatting. Bold. Italics. Superscripts. Strikethroughs. The visual cues humans rely on every time we read a doc, and on…

model-releasesjerry-liu--x

Let's talk document formatting. Bold. Italics. Superscripts. Strikethroughs. The visual cues humans rely on every time we read a doc, and ones existing OCR benchmarks completely ignore. 😱'199' struck

Safety

Preserving Long-Tailed Expert Information in Mixture-of-Experts Tuning

arXiv:2604.23036v1 Announce Type: cross Abstract: Despite MoE models leading many benchmarks, supervised fine-tuning (SFT) for the MoE architectures remains difficult because its router layers are fra

safetyarxiv-cs-cl

StratRAG: A Multi-Hop Retrieval Evaluation Dataset for Retrieval-Augmented Generation Systems

arXiv:2604.22757v1 Announce Type: cross Abstract: We introduce StratRAG, an open-source retrieval evaluation dataset for benchmarking Retrieval-Augmented Generation (RAG) systems on multi-hop reasonin

FlashAttention-4 gives the NVIDIA Blackwell platform its most optimized attention kernel yet

model-releaseslambda-labs

On March 5, 2026, the much-anticipated paper for FlashAttention-4 (FA4) was published. The code was dropped on GitHub months ago; early benchmarks circulated, and preliminary results were presented at

Holo360D: A Large-Scale Real-World Dataset with Continuous Trajectories for Advancing Panoramic 3D Reconstruction and Beyond

model-releasesarxiv-cs-cv

arXiv:2604.22482v1 Announce Type: new Abstract: While feed-forward 3D reconstruction models have advanced rapidly, they still exhibit degraded performance on panoramas due to spherical distortions. Mo

Navigating Large-Scale Document Collections: MuDABench for Multi-Document Analytical QA

arXiv:2604.22239v1 Announce Type: cross Abstract: This paper introduces the task of analytical question answering over large, semi-structured document collections. We present MuDABench, a benchmark fo

TRACE: Topology-aware Reconstruction of Accidents in CARLA for AV Evaluation

model-releasesarxiv-cs-ro

arXiv:2604.22068v1 Announce Type: cross Abstract: Validating Autonomous Vehicles (AVs) requires exposure to rare, safety-critical scenarios, infrequent in routine driving data. Existing benchmarks add

Wiggle and Go! System Identification for Zero-Shot Dynamic Rope Manipulation

arXiv:2604.22102v1 Announce Type: cross Abstract: Many robotic tasks are unforgiving; a single mistake in a dynamic throw can lead to unacceptable delays or unrecoverable failure. To mitigate this, we

We need more evals for document understanding. ParseBench is a really great start. I respect @llama_index ‘s work on this. 📈

model-releasesjerry-liu--x

We need more evals for document understanding. ParseBench is a really great start. I respect @llama_index ‘s work on this. 📈 We benchmarked GPT-5.5 on document understanding 📄📊 We ran it through Parse

26 Apr 2026

AUDITA: A New Dataset to Audit Humans vs. AI Skill at Audio QA

arXiv:2604.21766v1 Announce Type: new Abstract: Existing audio question answering benchmarks largely emphasize sound event classification or caption-grounded queries, often enabling models to succeed

Can MLLMs 'Read' What is Missing?

arXiv:2604.21277v1 Announce Type: new Abstract: We introduce MMTR-Bench, a benchmark designed to evaluate the intrinsic ability of Multimodal Large Language Models (MLLMs) to reconstruct masked text d

Context Is What You Need: The Maximum Effective Context Window for Real World Limits of LLMs

arXiv:2509.21361v2 Announce Type: replace-cross Abstract: Large language model (LLM) providers boast big numbers for maximum context window sizes. To test the real world use of context windows, we 1)

DAVIS: OOD Detection via Dominant Activations and Variance for Increased Separation

model-releasesarxiv-cs-cv

arXiv:2601.22703v2 Announce Type: replace Abstract: Detecting out-of-distribution (OOD) inputs is a critical safeguard for deploying machine learning models in the real world. However, most post-hoc d

Learning to Communicate: Toward End-to-End Optimization of Multi-Agent Language Systems

arXiv:2604.21794v1 Announce Type: new Abstract: Multi-agent systems built on large language models have shown strong performance on complex reasoning tasks, yet most work focuses on agent roles and or

Multilingual and Domain-Agnostic Tip-of-the-Tongue Query Generation for Simulated Evaluation

arXiv:2604.21096v1 Announce Type: cross Abstract: Tip-of-the-Tongue (ToT) retrieval benchmarks have largely focused on English, limiting their applicability to multilingual information access. In this

Symbolic Grounding Reveals Representational Bottlenecks in Abstract Visual Reasoning

arXiv:2604.21346v1 Announce Type: new Abstract: Vision--language models (VLMs) often fail on abstract visual reasoning benchmarks such as Bongard problems, raising the question of whether the main bot

TRAVELFRAUDBENCH: A Configurable Evaluation Framework for GNN Fraud Ring Detection in Travel Networks

arXiv:2604.21093v1 Announce Type: cross Abstract: We introduce TravelFraudBench (TFG), a configurable benchmark for evaluating graph neural networks (GNNs) on fraud ring detection in travel platform g

Wiring the 'Why': A Unified Taxonomy and Survey of Abductive Reasoning in LLMs

arXiv:2604.08016v2 Announce Type: replace Abstract: Regardless of its foundational role in human discovery and sense-making, abductive reasoning--the inference of the most plausible explanation for an

Measuring the Machine: Evaluating Generative AI as Pluralist Sociotechical Systems

arXiv:2604.20545v1 Announce Type: new Abstract: In measurement theory, instruments do not simply record reality; they help constitute what is observed. The same holds for generative AI evaluation: ben

Same Content, Different Answers: Cross-Modal Inconsistency in MLLMs

arXiv:2512.08923v2 Announce Type: replace Abstract: We introduce two new benchmarks REST and REST+ (Render-Equivalence Stress Tests) to enable systematic evaluation of cross-modal inconsistency in mul

researcharxiv-cs-ai

Agents

Thanks to @lmsysorg ！ Try it on SGLang now!🚀🚀

Thanks to @lmsysorg ！ Try it on SGLang now!🚀🚀 🚀 Qwen3.6-27B is here, and we have day 0 support on SGLang ✅ 27B params, beats Qwen3.5-397B-A17B across major coding benchmarks → Agentic coding → Text +

agentsqwen--x

Tokenised Flow Matching for Hierarchical Simulation Based Inference

arXiv:2604.20723v1 Announce Type: cross Abstract: The cost of simulator evaluations is a key practical bottleneck for Simulation Based Inference (SBI). In hierarchical settings with shared global para

AutomationBench

arXiv:2604.18934v1 Announce Type: new Abstract: Existing AI benchmarks for software automation rarely combine cross-application coordination, autonomous API discovery, and policy adherence. Real busin

Compile to Compress: Boosting Formal Theorem Provers by Compiler Outputs

arXiv:2604.18587v1 Announce Type: cross Abstract: Large language models (LLMs) have demonstrated significant potential in formal theorem proving, yet state-of-the-art performance often necessitates pr

Guys, I am absolutely astounded. The Qwen 3.6 27b is like a jump to Qwen 4 from Qwen 27B 3.5. I just did a full suite of front end design te…

model-releasesclem-delangue--x

Guys, I am absolutely astounded. The Qwen 3.6 27b is like a jump to Qwen 4 from Qwen 27B 3.5. I just did a full suite of front end design tests and agentic benchmarks, made entirely by it. VERDICT: Th

LiveVLN: Breaking the Stop-and-Go Loop in Vision-Language Navigation

model-releasesarxiv-cs-ro

arXiv:2604.19536v1 Announce Type: new Abstract: Recent navigation systems achieve strong benchmark results, yet real-world deployment often remains visibly stop-and-go. This bottleneck arises because

LLM-as-Judge Framework for Evaluating Tone-Induced Hallucination in Vision-Language Models

arXiv:2604.18803v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) are increasingly deployed in settings where reliable visual grounding carries operational consequences, yet their behavi

Lost in Translation: Do LVLM Judges Generalize Across Languages?

arXiv:2604.19405v1 Announce Type: new Abstract: Automatic evaluators such as reward models play a central role in the alignment and evaluation of large vision-language models (LVLMs). Despite their gr

Mango: Multi-Agent Web Navigation via Global-View Optimization

arXiv:2604.18779v1 Announce Type: new Abstract: Existing web agents typically initiate exploration from the root URL, which is inefficient for complex websites with deep hierarchical structures. Witho

Multi-modal Test-time Adaptation via Adaptive Probabilistic Gaussian Calibration

arXiv:2604.19093v1 Announce Type: cross Abstract: Multi-modal test-time adaptation (TTA) enhances the resilience of benchmark multi-modal models against distribution shifts by leveraging the unlabeled

RepIt: Steering Language Models with Concept-Specific Refusal Vectors

arXiv:2509.13281v5 Announce Type: replace Abstract: Current safety evaluations of language models rely on benchmark-based assessments that may miss localized vulnerabilities. We present RepIt, a simpl

Revisiting RaBitQ and TurboQuant: A Symmetric Comparison of Methods, Theory, and Experiments

arXiv:2604.19528v1 Announce Type: cross Abstract: This technical note revisits the relationship between RaBitQ and TurboQuant under a unified comparison framework. We compare the two methods in terms

Towards Optimal Agentic Architectures for Offensive Security Tasks

arXiv:2604.18718v1 Announce Type: cross Abstract: Agentic security systems increasingly audit live targets with tool-using LLMs, but prior systems fix a single coordination topology, leaving unclear w

Xpertbench: Expert Level Tasks with Rubrics-Based Evaluation

arXiv:2604.02368v4 Announce Type: replace Abstract: As Large Language Models (LLMs) exhibit plateauing performance on conventional benchmarks, a pivotal challenge persists: evaluating their proficienc

An Integrated Deep-Learning Framework for Peptide-Protein Interaction Prediction and Target-Conditioned Peptide Generation with ConGA-PePPI and TC-PepGen

model-releasesarxiv-cs-lg

arXiv:2604.18467v1 Announce Type: new Abstract: Motivation: Peptide-protein interactions (PepPIs) are central to cellular regulation and peptide therapeutics, but experimental characterization remains

21 Apr 2026

BhashaSutra: A Task-Centric Unified Survey of Indian NLP Datasets, Corpora, and Resources