Search: “benchmark”

MAPLE: Metadata Conditioned LLM Pretraining for Locale-Aware Question Answering

arXiv:2601.15236v2 Announce Type: replace Abstract: Large language models can memorize competing locale-specific facts yet fail to select among them when the locale changes, defaulting instead to a si

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Not Just Reason, Not Just Scan: Reinforcement Learning for Proactive Scientific Error Verification over Academic Paper

DGX agent

arXiv:2608.26596v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) are increasingly capable scientific assistants, yet they remain far from fully autonomous research. This transi

model-releasesarxiv-cs-cl

28 Aug 2026

Omni-Interactive Universal Embedder

arXiv:2608.27044v1 Announce Type: new Abstract: Multimodal representation learning has been shifting from traditional two-tower architectures to large language model (LLM)-based embedders due to their

PredVLA: A Sub-Million-Parameter Predictive-Coding Policy for Robot Manipulation

model-releasesarxiv-cs-ro

arXiv:2608.26673v1 Announce Type: new Abstract: Large pretrained vision-language-action models dominate modern robot-manipulation benchmarks, but it remains unclear how much model scale is necessary f

SLM-Conditioned Hierarchical Relation Routing for Labeled Property Graph Learning

arXiv:2608.26132v1 Announce Type: cross Abstract: Labeled property graphs combine relational structure with heterogeneous textual and categorical properties attached to both nodes and relationships. C

Temporally-Grounded Language Generation: Towards Real-Time Vision-Language Models

arXiv:2505.11326v2 Announce Type: replace-cross Abstract: Vision-language models (VLMs) have shown remarkable progress in offline tasks such as image captioning and video question answering. However,

A General-Purpose Molecular Foundation Model Transfers Across Diverse Olfactory Tasks

model-releasesarxiv-cs-lg

arXiv:2608.25893v1 Announce Type: new Abstract: Foundation models have transformed molecular property prediction, yet it remains unclear whether a molecular foundation model, fine-tuned on a single ca

Applications

EduDial: Constructing a Large-scale Multi-turn Teacher-Student Dialogue Corpus

arXiv:2510.12899v3 Announce Type: replace Abstract: Recently, several multi-turn dialogue benchmarks have been proposed to evaluate the conversational abilities of large language models (LLMs). As LLM

applicationsarxiv-cs-cl

FrontierChallenge: Evaluating Scientific Workflow Completion

arXiv:2608.24979v1 Announce Type: cross Abstract: Scientific agents increasingly analyze data, execute code, and produce research artifacts, yet most benchmarks emphasize final answers, isolated progr

Research

HealthBench-Psych: A Mental Health Subset of OpenAI's HealthBench

arXiv:2608.25071v1 Announce Type: new Abstract: General-purpose health benchmarks increasingly anchor claims about LLM medical performance, but they are not always resolved by clinical specialty, maki

researcharxiv-cs-cl

Mind2Report: Expert-Level Commercial Report Synthesis via Cognitive Deep Research Agent

arXiv:2601.04879v2 Announce Type: replace Abstract: Synthesizing informative commercial reports from massive and noisy web sources is critical for high-stakes business decisions. Although recent deep

PlanSightRAG: A Visual-First Multimodal RAG for Automating Question Answering and Compliance Checking for Civil Standard Plans

arXiv:2608.26091v1 Announce Type: cross Abstract: Civil infrastructure compliance checking has long relied on engineers manually reading legacy 2D plans; however, OCR-based automation strips away the

Reconstructing the Right Episode: Evaluating Interleaved Conversational Memory Beyond Long Context

arXiv:2608.25655v1 Announce Type: new Abstract: Conversations with chat assistants increasingly span many topics in a single long-running thread, challenging memory systems. Existing long-context and

SciMIF: Understanding Multimodal Instruction Following in Scientific Domains

model-releasesarxiv-cs-lg

arXiv:2608.25973v1 Announce Type: cross Abstract: Understanding instruction-following capabilities in scientific domains is essential for effectively leveraging Multimodal Large Language Models (MLLMs

Unfolding Scientific Papers into Multi-Turn Generation Trajectories for Continued Pre-Training

arXiv:2608.25826v1 Announce Type: new Abstract: A recent line of synthetic-data work reconstructs the thinking behind existing text rather than rewriting the text itself, but it operates on short web

Zero-WAM: In-Context World-Action Modeling from Human Videos for Open-Ended Task Generalization

arXiv:2608.26103v1 Announce Type: cross Abstract: Zero-shot cross-task generalization, where a policy must execute manipulation tasks never seen during training, remains a central challenge in robot l

Comparing Uncertainty Measurement and Mitigation Methods for Large Language Models: A Systematic Review

arXiv:2504.18346v4 Announce Type: replace-cross Abstract: Large Language Models (LLMs) have been transformative across many domains. However, hallucination, i.e., confidently outputting incorrect info

Evaluating Multiple LLM Generations with Validated Task Coverage

arXiv:2608.24228v1 Announce Type: new Abstract: Many LLM applications are most useful when they provide several candidate outputs for comparison, validation, or combination. Predominant evaluation set

More GPUs or a Smaller Cache? Tensor Parallelism versus KV Compression for Memory-Bound LLM Serving

arXiv:2608.23962v1 Announce Type: new Abstract: When an LLM serving deployment runs out of KVcache room, there are two well-established ways out. Tensor parallelism shards the weights and the KV cache

TangramPuzzle: Evaluating Multimodal Large Language Models with Compositional Spatial Reasoning

arXiv:2601.16520v2 Announce Type: replace-cross Abstract: Multimodal Large Language Models (MLLMs) have achieved remarkable progress in visual recognition and semantic understanding, yet precise compo

ToolRobustBench: Stage-Wise Perturbation Evaluation and Failure Diagnosis for Tool-Calling Agents

arXiv:2608.23635v1 Announce Type: cross Abstract: Large language models (LLMs) rely on tool calling as a fundamental agent capability, enabling them to invoke external systems and complete tasks beyon

Underrated Muse Glimmer

Benchmarked qwen3.8 xhigh, medium and muse glimmer. Xhigh effort mode with qwen3.8 took almost 30hrs. (And still failed on 16 cases because of the 32K output token limit) Medium effort mode and muse g

EviSafe: Evidence-Grounded Safety Evaluation for Vision-Language Models

arXiv:2608.23313v1 Announce Type: new Abstract: Vision-language model safety benchmarks typically evaluate only final responses: whether a model refuses, warns, or complies. This outcome-level view ca

Mac Studio M5 Max Cost Analysis

At $10k, you could get - 6.2B tokens with Qwen 3.8 Max (Qwen Pro plan) - 5.7B tokens with DeepSeek V4 Pro OpenRouter - 100B tokens with DeepSeek V4 Flash OpenRouter As a firm believer of local inferen

MedReaMM: Evaluating Large Multimodal Models on Expert-Level Clinical Diagnostic Synthesis

arXiv:2608.22323v1 Announce Type: new Abstract: The application of Large Language Models (LLMs) to diagnostic decision-making has garnered growing interest. However, existing benchmarks largely focus

What's the Catch? Evaluating Temporal Consistency in Vision-Language Models

arXiv:2608.23474v1 Announce Type: cross Abstract: Vision-language models (VLMs) achieve strong performance on video and image-sequence benchmarks, yet it remains unclear whether they capture temporal

Hardware

AgentX - InferenceXv3: Does CUDA Moat Hold up in Agentic Inferencing?

AgentX 1.0 is the first fully open‑source multi‑turn agentic coding inference benchmark (Apache 2.0), released August 2026 with a 1 million‑token context and over 1 million tokens of dataset built at

hardwaresemianalysis

Beyond Raw Transcripts: Structured Persona Extraction for LLM-Based Digital Twins

arXiv:2608.20344v1 Announce Type: new Abstract: LLM-based 'digital twins' aim to simulate how an individual would behavein new environments or respond to novel questions, given some representation of

Can Legal AI Know When It Is Wrong? And Do Students Know When It Is?

arXiv:2608.21089v1 Announce Type: new Abstract: Integrating Large Language Models (LLMs) into the Indian judiciary promises access to justice but introduces severe risks. We identify the 'inertia of c

Can Scientific Claims Be Removed from Large Language Models? A Systematic Evaluation of Claim-Level Unlearning

arXiv:2608.20960v1 Announce Type: new Abstract: Language models (LMs) are trained on static scientific corpora, whereas scientific knowledge continuously evolves through correction and revision. Scien

Do SpeechLMs Hear Their Own Opinions? Diagnosing and Mitigating Previous-Belief Contamination in Streaming Emotion Understanding

arXiv:2608.20769v1 Announce Type: cross Abstract: Streaming emotion understanding uses historical state while continuously interpreting current audio, often feeding the model's previous prediction bac

Free-Text Evaluation of LLMs for 5G Domain Knowledge and Fault Analysis using LLM-as-Judge

arXiv:2608.21021v1 Announce Type: cross Abstract: Real-world fault analysis in 5G and emerging 6G networks demands domain expertise to analyze free-text diagnostics, including root-cause explanations

Hardware

How NVIDIA Groq 3 LPX Unlocks Ultrafast Interactivity at Long Context on NVIDIA Vera Rubin

NVIDIA Groq 3 LPX, when paired with the Vera Rubin NVL72 platform, delivers a record‑setting 3,431 output tokens/second on the Artificial Analysis 100K context benchmark using Gemma 4 31B, demonstrati

hardwarenvidia-developer

Human-JEPA: A Human-Centric Vision Model that Perceives and Anticipates

arXiv:2608.21160v1 Announce Type: new Abstract: Machines that understand humans should perceive the present and anticipate the future. Existing human-centric vision model are pretrained on human image

KoViDoRe: Korean Visual Document Retrieval

arXiv:2608.20840v1 Announce Type: cross Abstract: Recent advances in multimodal retrieval have improved the ability to retrieve information from visually rich documents such as PDFs and reports. Howev

Natural-Language-Guided Generator-Agnostic Shortlisting for Protein Binder Design

arXiv:2608.20755v1 Announce Type: new Abstract: Modern de novo design workflows generate many candidate protein binders, but wet-lab validation capacity remains limited, making shortlisting a major bo

Qwen 3.8 27B Aider score

I ran the Aider benchmark on Qwen 3.8 27B FP8 with FP8 KV cache 256K context vLLM. The score: 72.9 This matches Gemini 2.5 Pro from 2025-04-12 which also scored 72.9. Beats Claude Opus 4 from 2025-05-

Real local agentic coding on a 12GB VRAM budget.

Thanks to Unsloth Dynamic 3.0 quants coming in slightly leaner and better preserved, I settled on Qwen 3.8 27B (`UD_Q4_K_XL`) at 100K context as my daily driver for Hermes Agent and OpenCode. On an RT

Has anyone actually made 64k feel like 300k+ with recursive local agents?

I'm running Qwen 3.8 27B locally on a single GPU. I can push the context to 131k, but I'd rather run it faster at 64k if the agent can manage context properly. What I have in mind is pretty simple: on

23 Aug 2026

A Strong Linear Baseline for Whole-Heart Cardiac Shape Completion on CT, with an Open Eleven-Structure Statistical Shape Model

arXiv:2608.19932v1 Announce Type: new Abstract: Public cardiac cohorts annotate different subsets of the heart, so shapes from separate sources cannot be pooled without shared correspondence. Among re

BeyondMasks: Evaluating Causal and Physical Consistency in Video Object Removal

arXiv:2608.20107v1 Announce Type: new Abstract: Recent advances in generative video models have significantly improved visual realism in video object removal, yet evaluation protocols still focus on m

Can Agent Memory Systems Track Evolving State?

arXiv:2608.19652v1 Announce Type: new Abstract: As LLM-based agents are deployed for longer and higher-stakes tasks, their memory systems continue to have crucial gaps. While existing memory benchmark

Fastest NVFP4 quant of Qwen3.8 27B out there

Here's a brand new Blackwell-native, prefill-optimized 4-bit quant that runs 50% faster on compatible hardware than a Q4 quant of the same memory footprint. And it runs 4-7% faster than other NVFP4 qu

GRACE: Grounded Reasoning via Adapter Composition and Evidence-Aware Calibration for Educational Visual Question Answering

arXiv:2608.19355v1 Announce Type: cross Abstract: Educational visual question answering, or VQA, requires models to solve curriculum-oriented multiple-choice questions using both language and visual e

Tutorials

Harvey is a great example of how American companies are building world-class specialized models: they took an open-source base (Kimi K3), po…

tutorialsclem-delangue--x

Harvey is a great example of how American companies are building world-class specialized models: they took an open-source base (Kimi K3), post-trained it on legal data, and delivered state-of-the-art

PersonalBench: Measuring the Authorship Gap in LLM Personalization

arXiv:2608.19746v1 Announce Type: new Abstract: Personalized text generation aims to make LLMs write in a specific individual's style, yet existing benchmarks measure task accuracy or preference align

Remember, Verify, or Ask? Cross-Family Evaluation of Memory Commitment in LLM Agents

arXiv:2608.19564v1 Announce Type: new Abstract: Persistent memory can personalize an LLM agent, but an incorrect durable update can silently distort future behavior. We study the memory-clarification

Rule-Compliant Visual Spatial Planning for Multimodal Large Language Models

arXiv:2608.20237v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) combine linguistic reasoning with visual perception, yet their ability to perform visual spatial planning under