Search: “benchmark”

Consistency-Guided Decoding with Proof-Driven Disambiguation for Three-Way Logical Question Answering

arXiv:2604.06196v1 Announce Type: cross Abstract: Three-way logical question answering (QA) assigns True/False/Unknown to a hypothesis H given a premise set S. While modern large language models

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

DISSECT: Diagnosing Where Vision Ends and Language Priors Begin in Scientific VLMs

DGX agent

arXiv:2604.06250v1 Announce Type: cross Abstract: When asked to describe a molecular diagram, a Vision-Language Model correctly identifies ``a benzene ring with an -OH group.'' When asked to reason ab

model-releasesarxiv-cs-ai

10 Apr 2026

Lost in Cultural Translation: Do LLMs Struggle with Math Across Cultural Contexts?

arXiv:2503.18018v2 Announce Type: replace Abstract: We demonstrate that large language models' (LLMs) mathematical reasoning is culturally sensitive: testing 14 models from Anthropic, OpenAI, Google,

SearchAD: Large-Scale Rare Image Retrieval Dataset for Autonomous Driving

arXiv:2604.08008v1 Announce Type: new Abstract: Retrieving rare and safety-critical driving scenarios from large-scale datasets is essential for building robust autonomous driving (AD) systems. As dat

TeamLLM: A Human-Like Team-Oriented Collaboration Framework for Multi-Step Contextualized Tasks

arXiv:2604.06765v1 Announce Type: cross Abstract: Recently, multi-Large Language Model (LLM) frameworks have been proposed to solve contextualized tasks. However, these frameworks do not explicitly em

TEMPER: Testing Emotional Perturbation in Quantitative Reasoning

arXiv:2604.07801v1 Announce Type: new Abstract: Large language models are trained and evaluated on quantitative reasoning tasks written in clean, emotionally neutral language. However, real-world quer

Tutorials

Understanding Task Transfer in Vision-Language Models

arXiv:2511.18787v2 Announce Type: replace Abstract: Vision-Language Models (VLMs) perform well on multimodal benchmarks but lag behind humans and specialized models on visual perception tasks like dep

tutorialsarxiv-cs-cv

Visually-grounded Humanoid Agents

arXiv:2604.08509v1 Announce Type: new Abstract: Digital human generation has been studied for decades and supports a wide range of real-world applications. However, most existing systems are passively

Weakly Supervised Distillation of Hallucination Signals into Transformer Representations

arXiv:2604.06277v1 Announce Type: new Abstract: Existing hallucination detection methods for large language models (LLMs) rely on external verification at inference time, requiring gold answers, retri

Industry

Do you understand what this means? For the first time, an Open Weight models is #1 on CyberSecutity. Sure there’s Mythos but we don’t have i…

The specific tweet from @0xSero is not directly accessible, and the search results don't surface the exact model or event being referenced in that post. However, based on the context clues in the t...

industryclem-delangue--x

9 Apr 2026

Tools

Loving the eval track! Super relevant stuff for @clawdbench Funny pic from own of the slides @aiDotEngineer @ibragim_bad @arafatkatze and ot…

A user on X (formerly Twitter) expressed enthusiasm for an evaluation ('eval') track at an AI Engineer (@aiDotEngineer) event, noting its relevance to ClawBench — an open-source benchmarking and ev...

toolsswyx--x

9 Apr 2026

Openness without compromises for your Apache Iceberg lakehouse

model-releasesgoogle-cloud-ai

Today, at the Apache Iceberg Summit in San Francisco, we are announcing the preview of read and write interoperability between BigQuery and Iceberg-compatible engines, including Trino, Spark, and othe

8 Apr 2026

Distributionally Robust Regularization of Sparse Integer Programming Trained Learning Models

arXiv:2311.01994v2 Announce Type: replace-cross Abstract: Building explainable machine learning models is crucial for human users to be able to interpret the proposed statistical relationship obtained

FlashAttention with Statistics: Low-Overhead Attention Features for Compact Hallucination Detectors

arXiv:2610.21483v1 Announce Type: new Abstract: Large language models can produce fluent answers that the supplied context does not support. A compact frozen proxy model can detect such answers by rea

Lethe: Link Inference Attacks For Evaluation of Edge Unlearning Methods

model-releasesarxiv-cs-lg

arXiv:2610.21869v1 Announce Type: new Abstract: Graph Unlearning (GU) aims to remove the influence of specific training data from a Graph Neural Network without retraining anew. While most methods add

MeddeX: A Retrieval-Augmented LLaMA-3 System with Gemini Vision for Evidence-Backed Medical Report Analysis

arXiv:2610.20632v1 Announce Type: new Abstract: Modern healthcare generates increasingly complex medical reports that challenge clinical professionals in terms of time and interpretive burden. Manual

Memory as Weights: Internalizing Long-Term History for Streaming Videos

arXiv:2610.22070v1 Announce Type: cross Abstract: Streaming vision-language models (VLMs) must continuously integrate information from the past while remaining responsive to current observations and u

Qwen 27B - sacrificing context in order run better quants, pros and cons.

model-releasesr-localllama

I was running Unsloth IQ4_XS for a while and while it provided me with a healthy 96K context on 20GB vram, I did notice all the issues it had during draft and all the time and local tokens wasted tryi

VELD-QA: A Large-Scale Dataset for Visual Evidence Grounding and Question Answering in Long Documents

arXiv:2610.20964v1 Announce Type: cross Abstract: Existing long-document QA benchmarks rarely combine large-scale human-authored questions with detailed visual evidence annotations, limiting diagnosis

VisualTrust: Video Reasoning under On-Screen Claim Framing

arXiv:2610.21159v1 Announce Type: new Abstract: On-screen text can describe what a video shows, or it can frame what the video is supposed to mean. Current benchmarks treat both alike, testing whether

WorkflowPerturb: Calibrated Stress Tests for Evaluating Multi-Agent Workflow Metrics

arXiv:2602.17990v4 Announce Type: replace Abstract: Multi-agent LLM systems that generate structured workflows from natural-language requests are now deployed in production across cloud automation, De

ACED-Bench: Evaluating How LLM Agents Acquire and Act on Causal Evidence

arXiv:2610.14381v1 Announce Type: new Abstract: Large language models are increasingly used as scientific assistants, yet we still lack reliable ways to evaluate whether they can investigate a system

Anuvani-Diarization: Zero Shot Speaker Diarization and Annotation Conventions Across 22 Languages

arXiv:2610.20327v1 Announce Type: new Abstract: India's 22 scheduled languages are spoken by over a billion people in conversations that mix languages, often within one turn, overlap freely, and take

Are We Really Measuring Forgetting? Pitfalls in Evaluating Machine Unlearning for Multimodal LLMs

arXiv:2610.19613v1 Announce Type: new Abstract: As machine unlearning for multimodal large language models (MLLMs) rapidly advances, rigorous evaluation of its effectiveness becomes increasingly impor

Agents

Choosing LLMs for Strategic Agent Interactions

arXiv:2610.18354v1 Announce Type: new Abstract: Game-theoretic benchmarks claim that large language model (LLM) strategic decision-making varies across game types, interaction structures, and capabili

agentsarxiv-cs-ai

ChronoCulture: Evaluating Cultural Awareness Across Time in Large Language Models

arXiv:2610.20472v1 Announce Type: new Abstract: Cultural attitudes and social norms evolve over time, yet most cultural benchmarks for large language models (LLMs) treat culture as a static attribute

Decomposability-Guided Cooperative Coevolution for Large-Scale Itinerary Planning

arXiv:2506.06121v3 Announce Type: replace Abstract: Large-scale itinerary planning is a variant of the traveling salesman problem, aiming to determine an optimal path that maximizes the collected poin

DivAlgo: A systematic evaluation of algorithmic diversity for code generation models

arXiv:2610.19157v1 Announce Type: cross Abstract: Code generation models are actively being used in professional and educational settings. However, the competence of large language models (LLMs) to ge

Do Factuality Methods Make LLMs More Factual? Reproducibility and Fair Comparison

arXiv:2610.18415v1 Announce Type: cross Abstract: Large Language Models (LLMs) are aligned to human preferences, after pre-training and supervised fine-tuning steps. However, standard RL alignment ste

Does the Linguistic Register of Thinking Matter? Wenyan as a Compact Reasoning Register for Chinese LLMs

arXiv:2610.19741v1 Announce Type: new Abstract: Visible reasoning traces of Chinese reasoning-oriented LLMs are typically written in modern vernacular Chinese, which may be verbose under limited decod

Evaluating LLMs as Agentic Workflow Executors in Real-World Customer Support Skills

arXiv:2610.14346v2 Announce Type: replace Abstract: Enterprise customer-support agents increasingly rely on Large Language Models (LLMs) to execute bounded, tool-using workflows. We study this problem

GesBench: When and Where Do Multimodal Language Models Perceive Social Meaning?

arXiv:2610.15129v1 Announce Type: new Abstract: Human social perception is incremental: we infer intent before an utterance is complete, revise our interpretation as new cues arrive, and attend to spe

How Do LLMs Understand Legal Philosophy? Diagnosing Legal-Philosophical Reasoning in LLMs

arXiv:2610.19427v1 Announce Type: new Abstract: Legal reasoning is a fundamental capability for large language models (LLMs) to complete legal AI tasks such as legal judgment prediction. In legal prac

Jev-ToM: Evaluating Theory of Mind in Typed Decision Models

arXiv:2610.18610v1 Announce Type: cross Abstract: Dialogue and collaborative assistants repeatedly judge what a partner knows, believes, or intends. Typed decision models offer a potential alternative

Kandinsky S2V 1.0: Audio-Driven Avatar Video Generation

arXiv:2610.19290v1 Announce Type: new Abstract: We present Kandinsky S2V 1.0, an open family of speech-to-video models for controllable audio-driven human animation, comprising Kandinsky S2V 1.0 Lite

MIRAGE: Metric Inference and Reliability-Aware Gaussian Estimation from Feed-Forward Vision

arXiv:2610.12568v2 Announce Type: replace Abstract: Feed-forward 3D priors enable high-quality reconstruction from image streams but often lack reliable metric scale and suffer from long-term trajecto

PXRD-Bench: How Close Are We to Determining Crystal Structures from Powder Diffraction?

arXiv:2610.16480v1 Announce Type: new Abstract: Determining crystal structures from powder X-ray diffraction (PXRD) would enable routine characterization of newly synthesized materials, including thos

QUASAR 4-bit QAT checkpoints for Qwen3.5, Gemma 4 and Muse Glimmer: closer to BF16 than Google’s and NVIDIA’s 4-bit releases

model-releasesr-localllama

Two months ago we released a fully quantized NVFP4 Qwen3.8-27B trained with QUASAR. We've since trained QUASAR checkpoints for a bunch of other recent models: Qwen3.5-4B, Gemma 4 E4B and 12B, and Muse

Tutorials

ReferFlow: Beyond Final-Target Localization for Complex Referring Expressions

arXiv:2610.20352v1 Announce Type: new Abstract: Multimodal Large Language Models have made significant progress in referring expression comprehension (REC), yet most REC benchmarks still focus on fina

tutorialsarxiv-cs-cv

RoTA-Bench: Evaluating 3D Spatial Reasoning through Interactive Projection Alignment

arXiv:2610.17665v2 Announce Type: new Abstract: Can vision-language models turn 2D observations into 3D understanding that supports precise action? This requires inferring object geometry, anticipatin

SAGE: Pushing 2-Bit Vector-Quantized LLMs toward Real-World Usability

arXiv:2610.14745v1 Announce Type: new Abstract: Extreme low-bit weight-only quantization is an effective approach to reducing the deployment costs of large language models. Below 3 bits, scalar quanti

Security in LLM-Assisted Coding: A Systematic Literature Review

arXiv:2610.15482v1 Announce Type: cross Abstract: Large Language Models accelerate software development but introduce severe security vulnerabilities. This systematic review investigates security risk

SPARC: Reliable Spatial Annotations from Robot Demonstrations at Scale

arXiv:2606.13497v2 Announce Type: replace-cross Abstract: This work introduces Spatial Annotations from Robot Demonstrations with Reliability Calibration (SPARC), a risk-aware framework that automatic

The Gap Is in the Geometry: A Convergent Meta-Analysis of Multi-Facet Aerial and Remote Sensing Video Understanding

arXiv:2610.19102v1 Announce Type: new Abstract: Aerial and remote sensing video is increasingly evaluated like web video, by posing multiple-choice questions to multimodal large language models (MLLMs

When 1+1 Backfires: Rethinking Relevance-Diversity Balancing for Sample-Aware Visual Token Reduction

arXiv:2610.16615v1 Announce Type: new Abstract: Visual token compression has emerged as a critical technique for reducing the inference cost of Vision-Language Models. Existing methods select tokens b

Conditional Multi-Event Temporal Grounding in Long-Form Video

arXiv:2606.15320v2 Announce Type: replace Abstract: Multimodal large language models have made rapid progress in video temporal grounding, yet real-world applications routinely require localizing ever

14 Oct 2026

From Rainfall to Road-Network Functionality: Transferable Graph Forecasting for Flood-Disrupted Transportation Networks

model-releasesarxiv-cs-lg

arXiv:2610.14622v1 Announce Type: new Abstract: Extreme rainfall can rapidly degrade road-network functionality (RNF), disrupting emergency mobility, accessibility, evacuation, logistics, and infrastr

14 Oct 2026

Game2Real-Bench: Do Toy Games Reveal and Transfer to Real-World Visual Reasoning?

arXiv:2610.18834v1 Announce Type: new Abstract: Visual reasoning is an important capability for vision-language models (VLMs) to understand visual evidence, spatial structure, constraints, and state c