Search: “benchmark”

MorphoOrgaAgent: A Foundation-Model-Based Multi-Agent System for Autonomous Organoid Analysis

arXiv:2609.08696v1 Announce Type: cross Abstract: Organoids are three-dimensional tissue models whose morphology provides important insights into tumor development, disease progression, and drug testi

9 Sept 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Qwen3-0.6B (400 MB) on a Samsung Note 8 (2017) phone drives a real desktop Chrome

DGX agent

Up front: I'm one of the people building the page-perception layer used here. We started by testing small local models. The result turned out to be more interesting than the original test. 12 small mo

model-releasesr-localllama

8 Sept 2026

Can Some Generous Soul Grant Me My Wish?

model-releasesr-localllama

Please, Oh gentle soul, please beautiful soul, please kindhearted soul, with serious hardware and time to kill, please benchmark Unsloth Qwen3.8-27B-Q8_K_XL against Qwen3.8-Next-Flash-Q4_K_XL and prov

7 Sept 2026

2x R9700, 64 GB DDR5 is an absolute beast machine with vLLM Radiance / R9V and Qwen 3.8 27b and Flash next

model-releasesr-localllama

I've been tinkering with local LLMs since the beginning of the year when I had an Intel Arc B580 and 32 GB of DDR5. Curiosity got the best of me and I bought the first R9700 about half a year ago, als

6 Sept 2026

Beyond Reproducibility: Towards Security-Aware Evaluation of Research Artifacts

arXiv:2605.06508v2 Announce Type: replace-cross Abstract: Research artifacts are widely shared to support reproducibility, and artifact evaluation (AE) has become common at many leading conferences. H

Contamination Inflates Scores but Rarely Reorders Large Language Model Leaderboards

arXiv:2609.02899v1 Announce Type: new Abstract: Benchmark contamination, the leakage of test items into training data, is widely described as a threat to the reliability of large language model (LLM)

LightEMMA: A Longitudinal Evaluation of Vision-Language Models for Autonomous Driving

arXiv:2505.00284v3 Announce Type: replace-cross Abstract: Rapid advances in vision-language models (VLMs) have generated growing interest in their application to autonomous driving. A prevailing assum

TabScope: Question-Adaptive Scope Selection for Table Question Answering

arXiv:2609.03395v1 Announce Type: cross Abstract: Large Language Models (LLMs) have shown strong performance on table question answering, yet their accuracy often degrades as table size increases. We

To What Extent Do Large Language Models Understand Bangla Idioms?

arXiv:2609.03410v1 Announce Type: new Abstract: Idiomatic expressions are an integral part of natural language, reflecting cultural nuances and posing unique challenges for computational models, parti

Very important paper accelerating research around AI discovery. https://x.com/omarsar0/status/2095858839534862664?s=20

Very important paper accelerating research around AI discovery. https://x.com/omarsar0/status/2095858839534862664?s=20 New benchmark from @Apodex_AI worth digging into. TRACES scores AI systems on dis

model-releasesdair-ai--x

Cite or Decline: A Strict Course-Grounded Chatbot for STEM Lecture Videos

arXiv:2609.01846v1 Announce Type: new Abstract: Recorded lecture videos, often enhanced with search and summarization features, are a standard study resource. However, students cannot easily ask cours

From Visual Cues to Spoken Narration: Rethinking Audio Description

arXiv:2609.01725v1 Announce Type: new Abstract: Audio Description (AD) provides spoken narration of visual events during dialogue gaps, making movies accessible to visually impaired audiences. The pro

Learning to Fuse LLMs with Ontology Rankers for Rare-Disease Diagnosis

arXiv:2609.02473v1 Announce Type: new Abstract: Ontology rankers remain useful for rare-disease diagnosis because each candidate can be traced to matched patient phenotypes. Large language models (LLM

Mediocrity is the key for LLM as a Judge Anchor Selection

arXiv:2603.16848v2 Announce Type: replace Abstract: The ``LLM-as-a-judge'' paradigm has become a standard method for evaluating open-ended generation. To address the quadratic scalability costs of pai

Rating the Raters: Rasch Measurement Theory for LLM Evaluation

arXiv:2608.27463v2 Announce Type: replace Abstract: LLMs now sit on every side of evaluation: as examinees scored on benchmarks, judges of other models' outputs, and raters of human-generated content.

Semantic Signal-Assisted Inspection and Recovery Allocation in Reverse Logistics

arXiv:2609.02116v1 Announce Type: new Abstract: Reverse-logistics operators often decide how to inspect and route returned assets before their condition is fully observed, while full inspection consum

Side note: when we released ARC-AGI-3 in March, and frontier models scored <1% on it, a few Singularitarian poasters took it as a personal i…

model-releasesfrancois-chollet--x

Side note: when we released ARC-AGI-3 in March, and frontier models scored <1% on it, a few Singularitarian poasters took it as a personal insult, and got very worked up about it. They argued the benc

CordisBench: Can Language Models Reason About Component Lifecycles in Dynamic Agent Harnesses?

arXiv:2609.01600v1 Announce Type: cross Abstract: Dynamic agent harnesses let language models change the software that shapes their own execution. This flexibility brings a new reasoning burden: a loc

Defense-as-Skill: Evolving Runtime Guard Skill for Skill-Augmented Agents

arXiv:2609.01487v1 Announce Type: cross Abstract: Skill-augmented agents load reusable skills as persistent runtime context, improving task performance but also giving malicious skills a durable chann

Disclosure-Gated User Simulation for Companion-Agent Evaluation

arXiv:2609.00982v1 Announce Type: cross Abstract: Using a large language model to play the user is now standard in scalable evaluation. It has a repeatedly diagnosed failure: the simulated user is exc

Dyn-3D: Unveiling and Resolving Ego-Motion Ambiguity in Vision-Language Models

arXiv:2609.01059v1 Announce Type: new Abstract: As Vision-Language Models (VLMs) tackle dynamic 3D spatial reasoning, ego-motion perception becomes essential to resolve monocular scale ambiguity. Howe

FinLifeBench: Exhaustive Life-Event History and Financial-State Reconstruction from Longitudinal Banking Dialogue

arXiv:2609.01198v1 Announce Type: new Abstract: Repeated banking interactions require assistants to maintain complete, current, and traceable customer records as life changes emerge incidentally in ro

HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?

arXiv:2609.01437v1 Announce Type: cross Abstract: As agents move from research prototypes to deployed tools, their capability increasingly depends on model-external execution infrastructure, commonly

OpenAgentFlow: Enabling System-Wide Safety Boundaries for Heterogeneous AI Agent Fleets

arXiv:2609.00015v1 Announce Type: new Abstract: AI agents powered by large language models are evolving from isolated assistants into heterogeneous systems in which multiple agents, planners, controll

Revealing Multi-View Hallucination in Large Vision-Language Models

arXiv:2603.23934v2 Announce Type: replace-cross Abstract: Large vision-language models (LVLMs) are increasingly being applied to multi-view image inputs captured from diverse viewpoints. Despite this

StudyBench: Can Self-Evolution Squeeze Textbooks for Olympiad Capability?

arXiv:2609.00787v1 Announce Type: new Abstract: Humans need to study only a handful of well-written textbooks to master a discipline and attempt its hardest problems. We argue that an ideal self-evolu

TempCloze: Can Video-LLMs Identify the Missing Middle?

arXiv:2609.01515v1 Announce Type: cross Abstract: Temporal reasoning benchmarks for Video-LLMs are often mediated by language, leaving room for linguistic shortcuts from option wording, answer correla

Where the Verifier Fails: A Category-Level Audit of Reward Signals in RLVR

arXiv:2609.01354v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) and standard benchmark evaluation both rely on an automatic verifier that turns a free text answer

AgenTRIM: Tool Risk Mitigation for Agentic AI

arXiv:2601.12449v2 Announce Type: replace-cross Abstract: AI agents are autonomous systems that combine LLMs with external tools to solve complex tasks. While such tools extend capability, improper to

Are LLMs Ready to Assist Physicians? PhysAssistBench for Interactive Doctor-Patient-EHR Assistance

arXiv:2606.18613v4 Announce Type: replace-cross Abstract: The most plausible near-term role of medical LLMs is to assist rather than replace physicians, yet current evaluations often test isolated cap

Beyond Semantic Similarity: Reducing Unnecessary API Calls via Behavior-Aligned Retriever

arXiv:2508.14323v3 Announce Type: replace Abstract: Tool-augmented LLMs invoke external functions to extend their capabilities, but errors in the invocation decision, such as calling a tool when none

Beyond the Payload: How User Invocation Shapes Coding Agent Vulnerability to Repository Poisoning

arXiv:2608.30686v1 Announce Type: cross Abstract: Coding agents are increasingly used for software engineering tasks, including bootstrapping projects from third-party repositories whose integrity can

E-Commerce Bench: Evaluating LLM Agents on Long-Horizon Autonomous Business Operation

arXiv:2608.30730v1 Announce Type: cross Abstract: Long-horizon agentic tasks go beyond chaining short tasks over more interaction turns. Their evolving dynamic environments and long-range dependencies

FractureFields: Contact-Aware Binary Multi-Field Transfer for Fractured 3D Gaussian Simulation

arXiv:2608.28982v1 Announce Type: new Abstract: Physics-integrated 3D Gaussian representations make it possible to simulate image-reconstructed assets directly as particles, but current Gaussia-MPM pi

HiRS-Agent: A Hierarchical Multi-Agent System for Reliable Long-Horizon Remote Sensing Task Solving

arXiv:2608.30672v1 Announce Type: new Abstract: Recent advances in large language models and multimodal models have pushed remote sensing (RS) processing from simple perception models to agentic syste

IndicDetect: Evaluating Cross-Lingual LLM-Generated Text Detection for Hindi, Telugu, and Tamil

arXiv:2608.29919v1 Announce Type: cross Abstract: The rapid proliferation of LLMs has further heightened the need to develop dependable AI-generated text detection, especially beyond English. Neverthe

More Capable, Less Faithful: A Multilingual Analysis of Mathematical (Un)Solvability Detection in LLMs

arXiv:2608.30463v1 Announce Type: new Abstract: Solvability detection is one of the most challenging aspects of mathematical reasoning for Large Language Models (LLMs). While prior work has studied th

nnMNet: Baseline for Martian Terrain Semantic Segmentation

arXiv:2608.29609v1 Announce Type: new Abstract: Semantic segmentation is a crucial task for understanding Mars, the most Earth-like planet in our solar system. However, it is challenging because the M

ReGraP-LLaVA: Reasoning enabled Graph-based Personalized Large Language and Vision Assistant

arXiv:2505.03654v3 Announce Type: replace-cross Abstract: Multimodal Large Language Models have shown strong performance across multimodal tasks, and recent personalized MLLMs can recognize user-speci

Reward-Oracle MCTS for Formal Theorem Proving: Sample-Efficient Search and the Need for Kernel-Level Proof Auditing

arXiv:2608.28639v1 Announce Type: new Abstract: Formal theorem proving with large language models remains challenging due to the difficulty of navigating large proof search spaces efficiently. Existin

Selection-Aware Stress Testing for Interactive Agents

model-releasesarxiv-cs-lg

arXiv:2608.30916v1 Announce Type: new Abstract: Agent evaluations often use one benchmark to choose a workflow and then search for task types where its advantage weakens, so both conclusions are selec

SVI2LoD3: Agent-Driven Reconstruction of LoD3 Facade Openings in Semantic 3D City Models from Volunteered Street View Imagery using Large Language and Visual Models

arXiv:2608.29992v1 Announce Type: new Abstract: This paper presents an end-to-end, agent-driven pipeline for the LoD3 reconstruction of facade openings in 3D city models, producing directly usable Cit

Towards Natural Personalization: Evaluating Long-Horizon Preference Following in Personalized User-LLM Interactions

arXiv:2603.04191v2 Announce Type: replace Abstract: Large Language Models (LLMs) are increasingly serving as personal assistants, where users may share individual preferences over extended interaction

ToxLens: A Reproducible Graph-Learning Framework for Leakage-Aware, Uncertainty-Calibrated Molecular Toxicity Prediction

model-releasesarxiv-cs-lg

arXiv:2608.30472v1 Announce Type: new Abstract: Molecular toxicity prediction is increasingly used to prioritise compounds before experimental testing, but conventional benchmark performance can overs

When Patients Cut In: Extending Clinical Conversational AI Safety to Interruptions

arXiv:2608.29241v1 Announce Type: new Abstract: Clinical voice agents are now deployed in routine care, where real patients do not wait their turn: they interrupt. These systems typically use a cascad

Loop engineering has emerged as a new skill for AI engineers But there is very little research measuring how effective it is. The best resul…

Loop engineering has emerged as a new skill for AI engineers But there is very little research measuring how effective it is. The best results on full tasks in a new benchmark is ~25%. LoopArena from

model-releasesdair-ai--x

31 Aug 2026

What Do Interaction Representations Actually Measure? Pre-Event Separability in Weakly-Supervised Violence Detection

arXiv:2608.27879v1 Announce Type: new Abstract: Articulated human pose provides detailed body-configuration information beyond coarse spatial relationships, but whether this detail yields greater disc

31 Aug 2026

Qwen3.8-27B thinking xhigh Vs. thinking off - Apple M5 Max