Knowledge catalogue
arXiv:2610.14146v1 Announce Type: new Abstract: Long-horizon terminal benchmarks measure whether an agent can sustain progress through extended workflows with process rewards, not only whether its fin
arXiv:2610.14570v1 Announce Type: new Abstract: Large language models (LLMs) support social simulation, but faithfully reproducing real users requires grounding both structured platform decisions and
I saw u/lkarlslund 's post just over a week ago regarding Ninfer6000 (a fork of Ninfer specifically tuned for 6000 blackwell and Qwen3.8-Flash-Next) and figured 'Hey I'm an idiot & bought one of those
arXiv:2610.15710v1 Announce Type: new Abstract: Long-term memory for assistants needs a gate that decides when a new utterance retires an old stored claim. Benchmark labels for this step mark a pair a
This was my first blog post at @llama_index! Go check out the cool diagrams and stuff Media Introducing ExtractBenchPro: the most challenging benchmark for real-world document extraction. We see a mas
arXiv:2610.14180v1 Announce Type: new Abstract: Organisations that want language-model agents for security work face a choice between open-weight models, which they can run and audit themselves, and f
arXiv:2610.14477v1 Announce Type: cross Abstract: Robustness tests for tool-using agents often change the agent's environment, for example by hiding a hostile note in an invoice, and then judge the ag
arXiv:2610.20195v1 Announce Type: new Abstract: Effective social interaction requires more than fluent responses: agents must reach plans that fit the participants' needs and circumstances. Existing e
arXiv:2610.15562v1 Announce Type: new Abstract: Large-scale video generation models are increasingly viewed as a promising path toward world models, yet their ability to capture causally coherent phys
arXiv:2610.17747v1 Announce Type: cross Abstract: Failure-attribution benchmarks ask an LLM judge which step or agent caused a failure. A high score should mean the judge checked the faulty step again
arXiv:2610.16170v1 Announce Type: new Abstract: Vision-language models (VLMs) can answer many geometric and physical reasoning questions correctly, yet still behave inconsistently in interaction, for
arXiv:2610.18694v1 Announce Type: new Abstract: Vision-language models (VLMs) struggle with metric and multi-view relationships that remain implicit in RGB observations. Geometry foundation models rec
arXiv:2610.18238v1 Announce Type: new Abstract: Vision-language models (VLMs) have been widely adopted for document parsing and optical character recognition (OCR), as they can accurately extract page
arXiv:2610.17685v1 Announce Type: cross Abstract: Enterprise agents face safety risks that cannot be determined from individual actions alone. Existing safety benchmarks largely evaluate action-local
arXiv:2610.16720v1 Announce Type: new Abstract: Existing benchmarks for GUI agents primarily focus on increasing task difficulty by extending task horizons. Besides, enabling agents to explore unfamil
muse wonder sota on hard document ocr tasks the more you know! We benchmarked Gemini 4 Argon, Muse Wonder, and Muse Spark comprehensively on hard document OCR tasks. They're all pretty good! - Muse Sp
arXiv:2603.13759v2 Announce Type: replace Abstract: Category-level multi-object trackers estimate trajectories for all detected instances, whereas some applications require propagating only targets sp
arXiv:2610.14583v1 Announce Type: cross Abstract: Autoregressive robot video world models based on discrete visual tokens remain unreliable over long rollout horizons. Retaining the entire generated h
arXiv:2610.18845v1 Announce Type: new Abstract: Earth observation (EO) models are still built by hand. For each new task, experts design a training recipe and tune it against a benchmark. We show that
arXiv:2610.16441v1 Announce Type: new Abstract: Current image-text evaluation often relies on a global score, which gives limited insight into which semantic details are supported by the image. Explai
arXiv:2610.14731v1 Announce Type: new Abstract: Long-term memory benchmarks for language agents primarily evaluate whether information from earlier interactions can be retrieved and used when relevant
arXiv:2610.13361v1 Announce Type: cross Abstract: As the production of software passes into the hands of software itself, coding agents are increasingly commissioned to build agents fitted to their us
arXiv:2610.13022v1 Announce Type: cross Abstract: As LLMs become increasingly capable of implementing software, a critical bottleneck emerges: whether they truly understand what users want. Users ofte
Build a reproducible benchmark, compare tool use across agents, and test whether a description change reduces unnecessary work while preserving outcomes. The post How to test agent tool descriptions w
arXiv:2506.09956v2 Announce Type: replace-cross Abstract: Indirect prompt injection attacks exploit a fundamental weakness of LLMs: executing unauthorized tasks from untrusted sources. Despite the pra
arXiv:2610.13534v1 Announce Type: new Abstract: Long-horizon embodied agents must maintain task-relevant information, coordinate heterogeneous robots, and adapt their decisions as execution unfolds. E
arXiv:2610.13583v1 Announce Type: new Abstract: Detectors of machine-generated text score almost perfectly on scientific-literature benchmarks, implying we can reliably distinguish with these detector
arXiv:2610.12667v1 Announce Type: new Abstract: Large language model (LLM) evaluations on clean benchmarks overstate model robustness: real inputs arrive perturbed, and these meaning-preserving varian
arXiv:2610.13172v1 Announce Type: new Abstract: World models enable large language model (LLM) agents to anticipate action effects and make informed decisions. For generalization, it is desirable for
arXiv:2610.13823v1 Announce Type: cross Abstract: Semantic query engines ask a large language model (LLM) to judge individual rows, and benchmarks compare these engines on the quality of a few executi
arXiv:2610.13910v1 Announce Type: new Abstract: Vision-Language Models (VLMs) increasingly analyze and reason about images, yet it remains unclear whether their internal representations faithfully enc
arXiv:2610.11889v1 Announce Type: cross Abstract: We present a benchmark for predicting final output and checkpoint state from source and input alone. It extends CRUXEval-style output prediction with
I built MaRN (Mapping Networks), a PyTorch library that lets you optimize a compact latent representation instead of directly training every model parameter. Some results from my current benchmarks: M
arXiv:2603.07980v2 Announce Type: replace-cross Abstract: As language models (LMs) evolve from chat assistants to long-horizon agents capable of multi-step reasoning and tool use, existing benchmarks
arXiv:2610.12402v1 Announce Type: cross Abstract: Existing spatial reasoning benchmarks mainly test spatial perception: reading off relations already visible in the input. Yet real-world spatial intel
arXiv:2610.09026v1 Announce Type: cross Abstract: We present BEACON-SP, an ontology-grounded Graph Retrieval-Augmented Generation (GraphRAG) framework for clinician-facing decision support in behavior
Grok 4.7 ranks first in legal matters Today we are announcing Harvey LAB-AA v1.1 in collaboration with Harvey. This updates our scoring methodology for the Legal Agent Benchmark (LAB) to add a halluci
arXiv:2610.09305v1 Announce Type: cross Abstract: Benchmarks for measuring the quality of action-conditioned world models are still evolving and shifting away from visual similarity-based metrics to a
arXiv:2610.09660v1 Announce Type: new Abstract: Automatic Short Answer Scoring (ASAS) requires models that can score student responses against question-specific criteria while remaining efficient and
arXiv:2610.07438v1 Announce Type: cross Abstract: Artifact removal routinely precedes the classification of electrodermal activity (EDA), on the assumption that a cleaner signal supports a better deci
arXiv:2610.07686v1 Announce Type: new Abstract: We introduce BluffJAX: an open-source suite of adversarial imperfect information games in JAX. We provide canonical implementations of games designed fo
arXiv:2610.08048v1 Announce Type: new Abstract: LLM agents often lack the operational knowledge to act reliably in new environments, as they must discover specific tool behaviors or environment conven
arXiv:2610.07981v1 Announce Type: new Abstract: Higher-order models (e.g., hypergraph neural networks) often outperform lower-order baselines on hypergraph learning benchmarks, and their advantages ar
arXiv:2610.08102v1 Announce Type: new Abstract: Conversational MLLM agents are increasingly expected to assist in professional workflows, from AI research and engineering design to product management
arXiv:2610.06986v1 Announce Type: new Abstract: AI research agents are increasingly used to search over programs, mathematical constructions, and proofs. However, existing systems typically optimize e
arXiv:2610.06910v1 Announce Type: new Abstract: Recent advances in Large Language Models (LLMs) have demonstrated remarkable capabilities in web front-end execution, with browser-based game generation
arXiv:2610.07937v1 Announce Type: new Abstract: Redpine Science gives models and agents a single access point to a wide range of peer-reviewed literature, queried directly through the Model Context Pr
arXiv:2610.07374v1 Announce Type: new Abstract: Omniprediction is a learning guarantee which requires a single predictor to be competitive relative to the best hypothesis from a benchmark class for an
arXiv:2610.08095v1 Announce Type: new Abstract: Natural-language access to RDF knowledge graphs is a core Semantic Web ambition. Large language models (LLMs) have advanced Text-to-SPARQL, yet on unfam
LIBERO is a robot manipulation benchmark: 130 tasks across five suites (spatial, object, goal, scene10, scene90), each with human demos and a language goal. It is the standard testbed for language-con
arXiv:2602.02140v2 Announce Type: replace Abstract: Recent advances in unified multimodal models (UMM) have demonstrated remarkable progress in both understanding and generation tasks. However, whethe
arXiv:2610.07792v1 Announce Type: cross Abstract: Large language model agents are increasingly deployed to perform complex tasks in real-world environments. However, the knowledge required for correct
arXiv:2610.08791v1 Announce Type: new Abstract: Video world models can produce visually convincing yet physically inconsistent sequences, raising concerns about their reliability for prediction and pl
arXiv:2610.06596v1 Announce Type: new Abstract: Short-wave infrared (SWIR) imaging has emerged as a promising modality for autonomous driving, yet its practical benefits over RGB remain poorly charact
arXiv:2610.05274v1 Announce Type: new Abstract: Roadside traffic reasoning requires every free-form textual claim to be backed by visual evidence. Existing grounded multimodal large language models (M
arXiv:2610.03826v1 Announce Type: new Abstract: Benchmark scores license claims about the capabilities being evaluated. We audit the inference licensed by an evaluation protocol, rather than the predi
arXiv:2610.04381v1 Announce Type: new Abstract: Fine-grained image editing requires more than producing a visually plausible result: an editor must execute the requested attribute change precisely whi
For their Mistral Large 4 release, @MistralAI ran a blind Surge human eval, with our expert software engineers evaluating five frontier models on coding quality. Le Chonk ranked #1 among open-weight m
arXiv:2610.06378v1 Announce Type: new Abstract: The rapid evolution of video generation has narrowed the perceptual gap between authentic and synthetic videos, making generalizable AI-generated video
arXiv:2610.05902v1 Announce Type: cross Abstract: 'Don't tune hyperparameters on the test set' is often stated in machine learning textbooks. Violating it is considered a cardinal sin that produces mi