Knowledge catalogue
arXiv:2610.19240v1 Announce Type: new Abstract: Financial disclosures express the underlying relationships of double-entry accounting through unstructured text. While financial LLM benchmarks predomin
arXiv:2610.15750v1 Announce Type: new Abstract: Google Earth AI is a family of geospatial AI models, together with an agent that orchestrates them. We present this agent, Google Earth AI's Geospatial
arXiv:2610.14071v1 Announce Type: cross Abstract: As LLM agents are increasingly deployed in high-stakes enterprise settings, understanding how they handle sensitive information in agentic dialogue ha
arXiv:2509.09658v3 Announce Type: replace Abstract: Hallucinations in multimodal large language models (MLLMs), where a model generates content inconsistent with the input image, pose significant risk
arXiv:2610.20548v1 Announce Type: new Abstract: Understanding the social intelligence of large language models (LLMs) is increasingly important for applications involving human-AI interaction and mult
arXiv:2610.17837v1 Announce Type: cross Abstract: Large language model (LLM) agents are increasingly used for vulnerability analysis, yet existing benchmarks provide limited coverage of exploitation t
arXiv:2610.16397v1 Announce Type: new Abstract: We study environment scaling as a way to increase agent-benchmark difficulty while holding core task objectives fixed. Our framework, Recursive Environm
arXiv:2610.17205v1 Announce Type: new Abstract: LLM-based decision-making systems, also called System One models, have recently gained popularity, most notably through Jev (Almeida, 2026). Given a typ
arXiv:2610.16634v1 Announce Type: cross Abstract: Recent Vision-Language-Action (VLA) models increasingly use natural-language instructions to condition autonomous driving behavior. Yet existing close
arXiv:2610.17079v1 Announce Type: new Abstract: Knowledge-based Visual Question Answering (VQA) requires reasoning that extends beyond visible image content, demanding external commonsense and factual
arXiv:2610.17986v1 Announce Type: new Abstract: A common practice in instance retrieval is to grow the gallery size to improve a benchmark's discrimination capability. One prominent example is ILIAS,
arXiv:2610.13152v1 Announce Type: new Abstract: Open coding agents are commonly trained through supervised fine-tuning on interaction trajectories whose final patches pass the benchmark test suite. Ho
arXiv:2610.12596v1 Announce Type: new Abstract: Agentic large language models now tackle complex tasks, yet strategic reasoning over long horizons is under-studied: most agentic benchmarks are short a
arXiv:2504.06185v3 Announce Type: replace-cross Abstract: Chronic wounds affect a large population, particularly the elderly and diabetic patients, who often exhibit limited mobility and co-existing h
This is a follow-up to my post from yesterday (https://www.reddit.com/r/LocalLLaMA/comments/1x2erdj/qwen3827b_on_a_single_3090_140_toks_on_code_with/) detailing the CUDA megakernel. Thank you all for
arXiv:2610.11458v1 Announce Type: cross Abstract: AI research progress can be viewed as the interaction between two processes: benchmark creation and method discovery. Historically, both were driven b
arXiv:2610.09079v1 Announce Type: cross Abstract: Large language models equipped with development environments have moved code generation toward repository-scale construction, yet building complete re
arXiv:2610.09163v1 Announce Type: new Abstract: Task-oriented conversational agents remain fragile under real world conversation scenarios as they rarely follow a predictable script, especially when u
arXiv:2610.08680v1 Announce Type: cross Abstract: Endpoint accuracy on abstract-reasoning benchmarks does not reveal whether a language model has acquired a transferable rule or fit distribution-speci
arXiv:2610.07338v1 Announce Type: cross Abstract: Audio benchmarks are built around short, pre-segmented clips, limiting model design to brief inputs or fixed vocabularies. To close this gap, we intro
Introducing Open SLM Evaluations! Open SLM Evaluations is a dataset containing benchmarks for 106 SLMs(150M parameters and below). It has benchmark scores for each model, links, parameter counts, and
arXiv:2605.06021v2 Announce Type: replace Abstract: Systematic reviews and meta-analyses often need numerical data reported only in figures, and extracting them with interactive digitisers usually req
arXiv:2609.33678v2 Announce Type: replace Abstract: We introduce SWE-Game, a benchmark of 247 tasks grounded in 41 executable reference Godot games spanning 13 gameplay categories in 2D and 3D. Five t
arXiv:2610.07309v1 Announce Type: new Abstract: Long-term-memory agents can retrieve relevant information that is inadmissible for the current request because it belongs to another principal, violates
arXiv:2610.06369v1 Announce Type: new Abstract: Integrating environmental sensor data with leaf imagery is widely reported to boost crop disease classification accuracy. In this work, we reveal that t
arXiv:2610.04139v1 Announce Type: new Abstract: Predicting future spatial states supports collision avoidance and timely decision-making in dynamic environments. However, existing vision-language mode
arXiv:2507.07297v2 Announce Type: replace Abstract: Recent advances in large vision-language models have led to impressive performance in visual question answering and multimodal reasoning. However, i
arXiv:2610.05878v1 Announce Type: new Abstract: Benchmarks expose vision-language-action (VLA) policies to few canonical instructions, while exhaustive deployment testing is impossible. We introduce O
arXiv:2610.05097v1 Announce Type: new Abstract: As multimodal large language models (MLLMs) reason for longer, attention to the initial visual input diminishes, weakening visual grounding. Visual memo
arXiv:2610.02627v1 Announce Type: new Abstract: An email assistant should not complete less work simply because a user phrases the same request differently. Yet most benchmarks test each task with onl
Automated lint: 19 errors, 64915 warnings, 3 info
arXiv:2610.00648v1 Announce Type: new Abstract: AI coding agents are ubiquitous in engineering workflows amongst industry and academia. Yet, despite their use in app coding, relatively less attention
arXiv:2610.01819v1 Announce Type: new Abstract: Benchmark gains are often mechanism-ambiguous: reproducing an improvement does not by itself identify why it occurs. We study finite-library mechanism d
arXiv:2610.00314v1 Announce Type: new Abstract: Research agents explain planned experiments. We measure predictive credit with paired forecasts sharing an intervention, forecaster, and outcome while v
arXiv:2610.02205v1 Announce Type: new Abstract: Programmable world models separate executable dynamics from visual generation, offering a promising foundation for next-generation game engines. However
arXiv:2610.00084v1 Announce Type: new Abstract: Detailed profession-specific system prompts raise token use and estimated cost per response without a consistent accuracy gain. We evaluate Scientific A
arXiv:2609.38948v1 Announce Type: cross Abstract: Frontier models excel at many digital benchmarks, yet their ability to drive a real car, an everyday human skill, remains largely untested. We present
arXiv:2609.39378v1 Announce Type: new Abstract: Real-world embodied tasks, from everyday activities to professional procedures, require agents to act under physical constraints while tracking evolving
arXiv:2609.39420v1 Announce Type: new Abstract: Large language models are strong general-purpose code generators, but executable algorithmic trading remains a demanding specialization target: a model
arXiv:2607.10891v2 Announce Type: replace Abstract: Large language models (LLMs) are rapidly shifting toward agents that solve tasks through diverse interfaces, including web and graphical user interf
arXiv:2609.38269v1 Announce Type: cross Abstract: Coding agents are increasingly asked to build software rather than patch it, yet benchmarks for from-scratch repository construction are mostly limite
arXiv:2609.36556v1 Announce Type: new Abstract: Recent studies report that LLM-based multi-agent systems (MAS) fail at rates of 41%-87%, yet to our knowledge, no benchmark to date supports systematic
arXiv:2609.36686v1 Announce Type: new Abstract: Identifying the root cause of an anomaly among hundreds of sensors is critical for preventing safety incidents and costly downtime in complex monitored
When scaling up agentic reinforcement learning (RL) and evaluation across massive parallel rollouts, frontier AI labs inevitably hit a bottleneck: Expensive GPU clusters sit idle, waiting minutes for
I benchmarked Qwen3-VL 8B Instruct (Q4_K_M, Ollama, M5 24GB, ~30s/doc) against Claude Opus 5.5, Sonnet 5 and GPT-5.6 Terra on: - receipts: CORD (Indonesia) and SROIE (Malaysia), 30 each - 20 scanned 1
TL;DR -- You should probably just use base Qwen3.6-35B, as only Occamy-1.0 is competitive with it. Tiel is a major let-down, worse than Ornith. KAT surprises (good), Nex surprises (bad). This post is
Cool paper on catching alignment failures with Jev. The ideas is to ask Jev one generic yes/no question about a model's response, and use its probability as a score. With no extra training, that score
Automated lint: 17 errors, 59669 warnings, 3 info
arXiv:2609.28560v1 Announce Type: cross Abstract: Evaluating a stochastic large language model is costly: benchmark scores estimate expected performance from randomized rollouts, yet uniform repetitio
arXiv:2609.28610v1 Announce Type: new Abstract: Benchmarking is an increasingly critical part of research in machine learning and the domains where it is applied, including healthcare. Yet, despite th
benched 5.5 is IN-SANE. It almost broke our benchmark. I thought it was a bug. Claude Opus 5.5 is the new #1 on our writing benchmark, and it is not close at all. 2631 Elo. Second one, Fable, is at 23
arXiv:2609.27867v1 Announce Type: new Abstract: A forecasting benchmark reports which method won. We show that the answer is set by the evaluator's choices before any model is fitted. We benchmark 24
In real-time streaming, every millisecond counts. For example, at 30 frames per second (fps), developers have a strict frame budget of just 33.3 ms (and only 16.6 ms at 60 fps) to ingest camera frames
arXiv:2609.25352v1 Announce Type: cross Abstract: Evaluation of large language models (LLMs) for safety, security, and privacy (SSP) relies heavily on static benchmarks, which suffer from score satura
I've been comparing and working on Gemma 4, Phi-4-Mini, and Qwen3-4B for local LLM use on an 8 GB Raspberry Pi 5. The results were indeed close, but each model has a different strength: Gemma 4 E2B: a
arXiv:2609.24971v1 Announce Type: new Abstract: Agents today often take real-world actions that depend on long-term memory and context recall over time. However, most current memory benchmarks are bui
Source: Jev Benchmarks Its training method is literally called 'Reinforcement Learning for Calibrated Decisions.' Calibration gap vs human labels (lower = better): Yes/no: Jev 5.0, Gemini 3.8 Flash 2.
https://preview.redd.it/nulsv53o8vqh1.png?width=4500&format=png&auto=webp&s=74765dbd409f4c221640f9f6000a685f6fdbb242 Spent weekend benchmarking the Splash engine (by Incoai) and extending its architec
Automated lint: 14 errors, 54840 warnings, 3 info
arXiv:2603.02540v2 Announce Type: replace Abstract: Large language models (LLMs) display a unified 'general factor' of capability across 10 benchmarks (a finding confirmed by our factor analysis of 15