Search: “benchmark”

I've been tinkering with local LLMs since the beginning of the year when I had an Intel Arc B580 and 32 GB of DDR5. Curiosity got the best of me and I bought the first R9700 about half a year ago, als

arXiv:2605.06508v2 Announce Type: replace-cross Abstract: Research artifacts are widely shared to support reproducibility, and artifact evaluation (AE) has become common at many leading conferences. H

arXiv:2609.02899v1 Announce Type: new Abstract: Benchmark contamination, the leakage of test items into training data, is widely described as a threat to the reliability of large language model (LLM)

arXiv:2505.00284v3 Announce Type: replace-cross Abstract: Rapid advances in vision-language models (VLMs) have generated growing interest in their application to autonomous driving. A prevailing assum

arXiv:2609.03395v1 Announce Type: cross Abstract: Large Language Models (LLMs) have shown strong performance on table question answering, yet their accuracy often degrades as table size increases. We

arXiv:2609.03410v1 Announce Type: new Abstract: Idiomatic expressions are an integral part of natural language, reflecting cultural nuances and posing unique challenges for computational models, parti

Very important paper accelerating research around AI discovery. https://x.com/omarsar0/status/2095858839534862664?s=20 New benchmark from @Apodex_AI worth digging into. TRACES scores AI systems on dis

arXiv:2609.01846v1 Announce Type: new Abstract: Recorded lecture videos, often enhanced with search and summarization features, are a standard study resource. However, students cannot easily ask cours

arXiv:2609.01725v1 Announce Type: new Abstract: Audio Description (AD) provides spoken narration of visual events during dialogue gaps, making movies accessible to visually impaired audiences. The pro

arXiv:2609.02473v1 Announce Type: new Abstract: Ontology rankers remain useful for rare-disease diagnosis because each candidate can be traced to matched patient phenotypes. Large language models (LLM

arXiv:2603.16848v2 Announce Type: replace Abstract: The ``LLM-as-a-judge'' paradigm has become a standard method for evaluating open-ended generation. To address the quadratic scalability costs of pai

arXiv:2608.27463v2 Announce Type: replace Abstract: LLMs now sit on every side of evaluation: as examinees scored on benchmarks, judges of other models' outputs, and raters of human-generated content.

arXiv:2609.02116v1 Announce Type: new Abstract: Reverse-logistics operators often decide how to inspect and route returned assets before their condition is fully observed, while full inspection consum

Side note: when we released ARC-AGI-3 in March, and frontier models scored <1% on it, a few Singularitarian poasters took it as a personal insult, and got very worked up about it. They argued the benc

arXiv:2609.01600v1 Announce Type: cross Abstract: Dynamic agent harnesses let language models change the software that shapes their own execution. This flexibility brings a new reasoning burden: a loc

arXiv:2609.01487v1 Announce Type: cross Abstract: Skill-augmented agents load reusable skills as persistent runtime context, improving task performance but also giving malicious skills a durable chann

arXiv:2609.00982v1 Announce Type: cross Abstract: Using a large language model to play the user is now standard in scalable evaluation. It has a repeatedly diagnosed failure: the simulated user is exc

arXiv:2609.01059v1 Announce Type: new Abstract: As Vision-Language Models (VLMs) tackle dynamic 3D spatial reasoning, ego-motion perception becomes essential to resolve monocular scale ambiguity. Howe

arXiv:2609.01198v1 Announce Type: new Abstract: Repeated banking interactions require assistants to maintain complete, current, and traceable customer records as life changes emerge incidentally in ro

arXiv:2609.01437v1 Announce Type: cross Abstract: As agents move from research prototypes to deployed tools, their capability increasingly depends on model-external execution infrastructure, commonly

arXiv:2609.00015v1 Announce Type: new Abstract: AI agents powered by large language models are evolving from isolated assistants into heterogeneous systems in which multiple agents, planners, controll

arXiv:2603.23934v2 Announce Type: replace-cross Abstract: Large vision-language models (LVLMs) are increasingly being applied to multi-view image inputs captured from diverse viewpoints. Despite this

arXiv:2609.00787v1 Announce Type: new Abstract: Humans need to study only a handful of well-written textbooks to master a discipline and attempt its hardest problems. We argue that an ideal self-evolu

arXiv:2609.01515v1 Announce Type: cross Abstract: Temporal reasoning benchmarks for Video-LLMs are often mediated by language, leaving room for linguistic shortcuts from option wording, answer correla

arXiv:2609.01354v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) and standard benchmark evaluation both rely on an automatic verifier that turns a free text answer

arXiv:2601.12449v2 Announce Type: replace-cross Abstract: AI agents are autonomous systems that combine LLMs with external tools to solve complex tasks. While such tools extend capability, improper to

arXiv:2606.18613v4 Announce Type: replace-cross Abstract: The most plausible near-term role of medical LLMs is to assist rather than replace physicians, yet current evaluations often test isolated cap

arXiv:2508.14323v3 Announce Type: replace Abstract: Tool-augmented LLMs invoke external functions to extend their capabilities, but errors in the invocation decision, such as calling a tool when none

arXiv:2608.30686v1 Announce Type: cross Abstract: Coding agents are increasingly used for software engineering tasks, including bootstrapping projects from third-party repositories whose integrity can

arXiv:2608.30730v1 Announce Type: cross Abstract: Long-horizon agentic tasks go beyond chaining short tasks over more interaction turns. Their evolving dynamic environments and long-range dependencies

arXiv:2608.28982v1 Announce Type: new Abstract: Physics-integrated 3D Gaussian representations make it possible to simulate image-reconstructed assets directly as particles, but current Gaussia-MPM pi

arXiv:2608.30672v1 Announce Type: new Abstract: Recent advances in large language models and multimodal models have pushed remote sensing (RS) processing from simple perception models to agentic syste

arXiv:2608.29919v1 Announce Type: cross Abstract: The rapid proliferation of LLMs has further heightened the need to develop dependable AI-generated text detection, especially beyond English. Neverthe

arXiv:2608.30463v1 Announce Type: new Abstract: Solvability detection is one of the most challenging aspects of mathematical reasoning for Large Language Models (LLMs). While prior work has studied th

arXiv:2608.29609v1 Announce Type: new Abstract: Semantic segmentation is a crucial task for understanding Mars, the most Earth-like planet in our solar system. However, it is challenging because the M

arXiv:2505.03654v3 Announce Type: replace-cross Abstract: Multimodal Large Language Models have shown strong performance across multimodal tasks, and recent personalized MLLMs can recognize user-speci

arXiv:2608.28639v1 Announce Type: new Abstract: Formal theorem proving with large language models remains challenging due to the difficulty of navigating large proof search spaces efficiently. Existin

arXiv:2608.30916v1 Announce Type: new Abstract: Agent evaluations often use one benchmark to choose a workflow and then search for task types where its advantage weakens, so both conclusions are selec

arXiv:2608.29992v1 Announce Type: new Abstract: This paper presents an end-to-end, agent-driven pipeline for the LoD3 reconstruction of facade openings in 3D city models, producing directly usable Cit

arXiv:2603.04191v2 Announce Type: replace Abstract: Large Language Models (LLMs) are increasingly serving as personal assistants, where users may share individual preferences over extended interaction

arXiv:2608.30472v1 Announce Type: new Abstract: Molecular toxicity prediction is increasingly used to prioritise compounds before experimental testing, but conventional benchmark performance can overs

arXiv:2608.29241v1 Announce Type: new Abstract: Clinical voice agents are now deployed in routine care, where real patients do not wait their turn: they interrupt. These systems typically use a cascad

Loop engineering has emerged as a new skill for AI engineers But there is very little research measuring how effective it is. The best results on full tasks in a new benchmark is ~25%. LoopArena from

arXiv:2608.27879v1 Announce Type: new Abstract: Articulated human pose provides detailed body-configuration information beyond coarse spatial relationships, but whether this detail yields greater disc

I ran a few benchmarks on my MacBook Pro M5 Max with oMLX: Running Qwen3.8-27B with thinking off has a big effect on output quality. Running it with thinking on xhigh burns 5.5x the tokens and runs 6x

Below are the benchmark results for running Qwen3.8-Flash-Next on Strix Halo using the Vulkan backend of llama.cpp, combined with MTP model. Hardware Item Details CPU AMD Ryzen AI MAX+ 395 (16C/32T) G