Search: “benchmark”

Stellar Colosseum: A Many-Agent Harness for Long-Horizon Research in Mathematics and Theoretical Computer Science

arXiv:2609.15983v1 Announce Type: new Abstract: Language models can produce plausible short proofs, but may still be unreliable on long-horizon research problems, where progress depends on a sequence

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Sweet Talkers: How Query Formulation Shapes Sycophancy in Romantic Relationship Advice

DGX agent

arXiv:2609.13841v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used for emotional support and relationship advice, where a model's tendency to preserve a user's face can

model-releasesarxiv-cs-cl

15 Sept 2026

The Router Within: Eliciting Native Skill Routing from a Frozen LLM

arXiv:2609.15982v1 Announce Type: cross Abstract: Skills extend an LLM agent beyond its parametric knowledge, and the gain they promise rests on picking the right one. Deployed harnesses route by prel

Towards a Mechanistic Understanding of Propositional Logical Reasoning in Large Language Models

arXiv:2601.04260v2 Announce Type: replace Abstract: Understanding how Large Language Models (LLMs) perform logical reasoning internally remains a fundamental challenge. While prior mechanistic studies

TwinICL: Diagnosing Multimodal In-Context Learning through Paired Counterfactuals

arXiv:2609.15028v1 Announce Type: new Abstract: In-context learning (ICL) enables models to infer tasks from demonstrations, but existing benchmarks generally lack matched text and image versions need

Uncertainty-Aware Calibrated Clinical Text Classification with Large Language Models

arXiv:2509.19375v2 Announce Type: replace-cross Abstract: Large language models are increasingly used for clinical text classification, where overconfident misclassifications can directly affect patie

V-ICAL Bench: Evaluating Video In-Context Learning for Multimodal Agents in Interactive Environments

arXiv:2609.15683v1 Announce Type: new Abstract: While In-Context Learning (ICL) enables models to adapt from exemplars without parameter updates, multimodal ICL remains largely underexplored, particul

Verifiable by Construction: Claim-Level Evaluation of Verbatim Citation in Clinical Question Answering

arXiv:2609.15964v1 Announce Type: new Abstract: Large language models (LLMs) have been widely adopted for clinical question answering (QA). Current systems can attach citations to their answers, but t

When Consistency Does Not Mean Reliability: Evaluating Local LLM Judges Against Human Ratings

arXiv:2609.13824v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used to evaluate the responses of other language models. This approach, known as LLM-as-a-Judge, is faster

K2 Horizon lineup is out on AA, and once again AA plots are misleading.

The full K2 Horizon lineup is out on Artificial Analysis. The AA intelligence vs. parameters plots show that - 0.9B and 375B are bad - 3.7B and 7B are SOTA - 36B A4B is SOTA for hardware with poor mem

14 Sept 2026

Physics as the label for measuring and correcting materials reasoning in multimodal models

arXiv:2609.12181v1 Announce Type: new Abstract: Vision-language and language models increasingly interpret materials data, yet benchmarks report that they hallucinate invalid properties and violate ph

14 Sept 2026

Semantically Aligned Gradient-Driven Context-Preserving Image Editing

arXiv:2609.12691v1 Announce Type: new Abstract: Instruction-guided image editing has a training-time blind spot. Generative editors are never required to semantically verify whether their outputs actu

14 Sept 2026

Dear 24G owners, try VLLM you might be able to run Qwen3.8 27B INT4, 144K FP8 KV on RTX 3090 with better speed. (TLDR VLLM AOT)

VLLM Benchmark: Prefill, Prompt processing - avg, 871.93 tok/s (3 hours constant running xhigh) - 10K prompt, 1000.26 tok/s (16 runs) - 90K prompt, 743,59 tok/s (16 runs) Decode, tok gen - avg, 38.39

13 Sept 2026

llm performance community metric

my question about LLM performance We see a lot of posts about token prediction, token generation per second, etc. But is it really the metric? I can see that DeepSeek V4 Flash 0731 (with DSPark; mac s

13 Sept 2026

COBRA-Skills: Contextual Bandit-Guided Evolution for Agent Skill Optimization

arXiv:2609.11682v1 Announce Type: new Abstract: Large language model (LLM) agents can benefit from reusable skills distilled from prior task experience, yet existing skill optimization methods often r

12 Sept 2026

DeepSeek V4.1 Flash on 8× A40: ~40 tok/s Q2_K and ~32 tok/s Q4_K_M with TensorSharp

Hey everyone — I’m building TensorSharp, an open-source LLM inference engine. Here are the latest DeepSeek V4.1 Flash GGUF results using its native ggml_cuda backend. Setup: 8× NVIDIA A40, layer split

12 Sept 2026

Making Alternative Data Work: Context-Augmented LLMs for Financial Forecasting

arXiv:2609.11607v1 Announce Type: new Abstract: When forecasting a firm's future financial performance, alternative data - data collected from non-traditional sources such as consumer transactions, we

12 Sept 2026

AgentAudit: An Open, Extensible Framework for Full-Lifecycle Trust Evaluation of AI Agents

arXiv:2609.09875v1 Announce Type: new Abstract: Existing evaluation frameworks mostly assess only one part of AI agents, such as task completion (AgentBench) or security robustness (AgentDojo, ASB), r

Local Ai

Beyond Word Error Rate: A Switch Aware Evaluation of ASR and Audio Language Models on English Yoruba Code-Switched Speech

arXiv:2609.11786v1 Announce Type: new Abstract: Automatic speech recognition (ASR) systems and audio language models (audio LMs) now report low error rates on monolingual benchmarks, but their behavio

local-aiarxiv-cs-cl

Enabling Knowledge Graph Understanding at Scale with the EXplore Your Graphs ENgine (EXYGEN)

model-releasesarxiv-cs-lg

arXiv:2609.11569v1 Announce Type: cross Abstract: We present EXYGEN (EXplore Your Graphs ENgine), a framework for knowledge graph (KG) understanding that enables conversational access to KGs at scale.

LLM-Anchored Paralinguistic Enrichment for Alzheimer's Disease Detection

arXiv:2609.10896v1 Announce Type: new Abstract: Speech-based automatic detection of Alzheimer's disease (AD) provides a non-invasive and scalable approach to early cognitive screening. AD affects both

R4Tun: LLM-guided adaptive segmental tunnel lining segmentation in point clouds

arXiv:2609.11360v1 Announce Type: new Abstract: Automated inspection of segmental tunnel linings requires adaptive segmentation from 3D point clouds, yet expert-tuned pipelines often degrade when tunn

RAG-Safety-Bench: Reliable Evaluation of Retrieval-Augmented LLM Safety

arXiv:2609.11758v1 Announce Type: new Abstract: Allowing large language models (LLMs) to retrieve information from a set of trusted documents can increase reliability and reduce hallucination. However

Recognizing Is Not Reversing: A Controlled Inversion Test of Fact-Preserving News Framing

arXiv:2609.11769v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used to analyze and rewrite news, yet current framing studies mainly evaluate generation, detection, or wh

Reinforcement learning for Quantum Tiq-Taq-Toe

arXiv:2411.06429v2 Announce Type: replace Abstract: Quantum Tiq-Taq-Toe is a well-known benchmark and playground for both quantum computing and machine learning. Despite its popularity, no reinforceme

Same Day, Same Story; One Day Ahead, a Different Signal: The Dual Validity of Financial Sentiment

arXiv:2609.11144v1 Announce Type: cross Abstract: Financial NLP has a standard workflow: validate a sentiment tool against human labels, then trust it to extract market signal. This assumes the two ev

5-Dialects-BN: Unmasking the Impact of Transliteration on Bangla Dialectal LLMs

arXiv:2609.09964v1 Announce Type: new Abstract: Large Language Models (LLMs) have achieved remarkable progress across natural language processing (NLP) tasks, yet their capabilities degrade sharply fo

AtomCite: Verification and Correction of Supplied Page-Level Citations in Multi-Page Documents

arXiv:2609.05802v1 Announce Type: cross Abstract: Large language models answering questions over multi-page documents are expected to cite the supporting pages, yet supplied citations are sometimes in

Beyond 'AI Helps Humans': Decision-Targeted Evaluation Design for Human-Agent Teams in the Agentic Era

arXiv:2609.05527v1 Announce Type: new Abstract: Wherever a coding agent works under engineer supervision, or a clinical model assists a radiologist, the deployment question is whether to keep the huma

Dynamics of meaning: Towards the Evaluation of Diachronic Semantic Change in Sinhala

arXiv:2609.08609v2 Announce Type: replace Abstract: Tracking semantic change in low-resource languages across extensive historical timelines presents significant challenges due to data scarcity and th

Evaluating the Scalability and Adversarial Generalization of GRPO-Trained NLI Models

arXiv:2504.18376v3 Announce Type: replace-cross Abstract: Natural Language Inference (NLI) is a central task in natural language understanding with applications in fact-checking, question answering, a

Research

FreqBLiMP: Frequency-Controlled Minimal Pairs Reveal Robustness and Fragility of LLMs Under Lexical Rarity

arXiv:2609.07153v1 Announce Type: cross Abstract: Minimal-pair benchmarks such as BLiMP evaluate linguistic knowledge by testing whether language models (LMs) prefer acceptable sentences over minimall

researcharxiv-cs-ai

ObGynLongBench: Revealing the Evidence-to-EHR Gap in Longitudinal EHR Decision-Making

arXiv:2609.07601v1 Announce Type: cross Abstract: The application of large language models (LLMs) to personalized medical assistants has garnered growing interest. However, existing medical benchmarks

Safety

Quantization Amplifies Determinism, Not Bias: Scale-Dependent Behavioral Effects of Serving-Time Weight Compression

arXiv:2609.07901v1 Announce Type: new Abstract: Weight quantization largely determines the economics of serving open-weight LLMs. Its costs are usually assessed with capability benchmarks, on which 4-

safetyarxiv-cs-ai

SCIRIGOR:Evaluating Open-Ended Scientific Analysis Beyond Final Scores

arXiv:2609.06192v1 Announce Type: new Abstract: Scientific coding agents produce interdependent code, results, figures, and claims, yet evaluating final outputs alone does not establish whether their

Applications

Solving versus Verifying: Catching Contradictions in Tax Reasoning Systems

arXiv:2609.05928v1 Announce Type: cross Abstract: Large language models now compute correct tax liabilities on over 90% of well-formed cases in statutory benchmarks, which makes them candidates for th

applicationsarxiv-cs-ai

The human-authorship halo: attribution bias in literary style evaluation by humans and AI

arXiv:2510.08831v2 Announce Type: replace Abstract: As AI writing tools become widespread, we need to understand how both humans and machines evaluate literary style, a domain where objective standard

TreeThink: A Modular Tree Search Library for Mathematical Reasoning with LLMs

arXiv:2607.11258v2 Announce Type: replace Abstract: Tree search algorithms enable systematic exploration of the proof space in neural theorem proving. Existing LLM tree search libraries primarily targ

Typed Federated Artifacts for the Agentic Web:Sharing Tool-Routing Knowledge Across Frozen,Heterogeneous LLM Agents

arXiv:2609.06815v1 Announce Type: cross Abstract: An open, networked web will allow agents to run frozen models from multiple vendors, keep their history private, and teach each other which tool to ca

We're Cooked! - Probing LLM Political Alignment Via Conflict-Framed Recipe Translation

arXiv:2609.07568v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly deployed for translation tasks, yet their implicit political positioning in such contexts remains unders

A Multimodal Label Forecasting Method for Aperiodic Visuo-Motor Time Series

model-releasesarxiv-cs-ro

arXiv:2609.07930v1 Announce Type: new Abstract: Deep learning models have been increasingly applied to Time Series Forecasting (TSF) in recent years. Transformer-based and MLP-based models have both b

BanglaMemeX: Advancing Cultural Metaphoric Image Interpretation in Bangla with a Multimodal Explainable Dataset

arXiv:2609.08029v1 Announce Type: cross Abstract: Vision Language Models have achieved strong performance on multimodal benchmarks, yet their ability to reason about culturally grounded and metaphor-r

CALIPER: Clean Scenes Cannot Rank Physical Inference in Pretrained Visual Representations

arXiv:2609.08250v1 Announce Type: cross Abstract: How far a pushed object slides depends on its mass and friction, which no single image reveals. Pretrained visual encoders are increasingly used as th

Charts Are Beyond Pixels: Probing for Layer-Wise Chart Understanding and Editing

arXiv:2609.08657v1 Announce Type: new Abstract: Charts are structured visual compositions whose elements have distinct functional roles, semantic correspondences, and visibility relations. This struct

From Synthetic Priors to Model Behavior: Structural Coverage in Tabular Foundation Models

model-releasesarxiv-cs-lg

arXiv:2609.06912v1 Announce Type: new Abstract: Tabular foundation models (TFMs) are commonly pretrained on large collections of procedurally generated synthetic tasks, yet it remains unclear how well

Multi-Grid Post-Training for Long-Form Multi-Shot Video Generation

arXiv:2609.06373v1 Announce Type: new Abstract: Generating long-form multi-shot videos requires coherent within-shot motion and visually consistent narratives across shots. Existing video generators f

Segment Any Motion with Radar: Robust Multimodal Moving-Object Segmentation and Tracking

arXiv:2609.08346v1 Announce Type: new Abstract: Moving-object perception must decide which image regions correspond to real motion and keep every instance identified over time. Methods that read motio

Social Intuition vs. Machine Reasoning: Anticipating Human-Robot Interaction from multiple modalities

arXiv:2609.07394v1 Announce Type: new Abstract: Anticipating whether a person will interact from one's own perspective is a highly intuitive task for humans, that relies on a combination of cues. We i