Knowledge catalogue
arXiv:2610.08303v1 Announce Type: new Abstract: Current evaluation of multilingual Large Language Models (LLMs) rests on an implicit Translation-Isomorphism Assumption (TIA): that semantic structures
OpenDocRouter is here! https://www.opendocrouter.ai/ Some time ago, I realized LlamaIndex was really good at benchmarking and applying nearly every model to document parsing (see: ParseBench). So, why
arXiv:2610.08691v1 Announce Type: new Abstract: Large language model agents are accelerating scientific automation, yet verified executions rarely become persistent program-level improvements, and exi
arXiv:2610.07763v1 Announce Type: new Abstract: The rapid progress of LLM-based multi-agent systems (MAS) has shown that they largely outperform single agents on coding, math, and QA tasks, where exec
Disclosure: This unit was provided free of charge by Orange Pi for testing. No editorial review, no preconditions, no script. All data, bottlenecks, and thermal behavior are reported directly from har
decision models + local ai = ❤️ tried Laya (421M) on llama.cpp in a little apple-picking benchmark, it's almost as good as Qwen 4B at 12x the speed. Media Decision models in llama.cpp are now availabl
arXiv:2610.06450v1 Announce Type: new Abstract: Foundation models (FMs) are increasingly being developed for general time series and physiological signals, yet their transferability to downstream phys
arXiv:2610.05790v1 Announce Type: new Abstract: Remote sensing foundation models (RSFMs) are commonly evaluated using aggregate metrics, which can hide systematic performance disparities across ecolog
arXiv:2510.13276v2 Announce Type: replace Abstract: The rapid advancement of long-context vision language models (LCVLMs) has led to a significant expansion of their context windows. However, an exten
arXiv:2610.04456v1 Announce Type: new Abstract: Deep learning models for crop leaf disease recognition routinely report near-perfect accuracy yet are typically trained and evaluated on a single datase
arXiv:2610.04155v1 Announce Type: cross Abstract: Open electromagnetic-transient datasets are beginning to make reproducible learning-based protection studies possible, but the practical use of these
arXiv:2610.05341v1 Announce Type: new Abstract: Chronic wound management affects over 8.2 million patients in the United States and imposes substantial clinical and economic burden. Clinical wound ass
arXiv:2610.03526v1 Announce Type: cross Abstract: Explainers for Graph Neural Networks (GNNs) are commonly evaluated by their plausibility, i.e., how well their explanations recover a predefined groun
arXiv:2610.02460v1 Announce Type: new Abstract: Recent benchmarks rely on user simulators to evaluate AI agents in multi-turn interaction. While existing simulation techniques demonstrate surface fide
arXiv:2610.03020v1 Announce Type: new Abstract: Long-term agents must remember not only what is true about a user, but also how a particular agent should work with that user as their shared history ev
arXiv:2610.02986v1 Announce Type: new Abstract: Membership inference on large language models (LLMs) aims to determine whether a given text sample was included in an LLM's training data, without acces
arXiv:2610.02982v1 Announce Type: new Abstract: Programmable logic controllers (PLCs) coordinate industrial equipment by reading sensor inputs and issuing control commands. Evaluating whether large la
We’re launching ReviewBench, a benchmark for code review agents built on representative GitHub pull requests, multi-source ground truth, calibrated evaluation, and production-aligned metrics. The post
arXiv:2610.02599v1 Announce Type: new Abstract: Sustainable protein discovery lacks the fast computational proxies, analogous to molecular docking or density functional theory, that accelerate drug an
arXiv:2610.02256v1 Announce Type: new Abstract: Electricity price forecasting (EPF) supports scheduling, bidding, and risk management in electricity markets, yet existing benchmarks focus mainly on nu
arXiv:2610.01488v1 Announce Type: cross Abstract: Backchannel prediction has been studied almost entirely in dyadic conversation. We introduce a multi-party benchmark based on the AMI corpus, comprisi
arXiv:2610.00682v1 Announce Type: new Abstract: Large language models (LLMs) increasingly underpin scientific AI applications that reason over structured knowledge, from biomedical question answering
arXiv:2610.02202v1 Announce Type: new Abstract: What makes great scientists great? Even as AI systems start to make progress on open problems, scientists remain far ahead of them at sensing which prio
arXiv:2610.00666v1 Announce Type: cross Abstract: Qualitative comparison figures are central evidence in computer vision papers, and vision-language models (VLMs) are increasingly used to judge them.
arXiv:2609.39215v1 Announce Type: cross Abstract: Time series anomaly detection (TSAD) is increasingly deployed in streaming settings, where data arrive sequentially and may exhibit non-stationarity.
arXiv:2605.06462v2 Announce Type: replace Abstract: Progress in graph learning is hindered by benchmark practices that conflate the contributions of node features and graph structure, making it hard t
arXiv:2609.39903v1 Announce Type: new Abstract: Scientific software presents a demanding test for computer-using agents based on visual language models (VLMs): completing a research workflow requires
arXiv:2609.38974v1 Announce Type: new Abstract: Large language models are reshaping ecommerce from static recommenders into interactive shopping assistants, yet real-world shopping requires session-le
arXiv:2609.39847v1 Announce Type: cross Abstract: Audio language models (ALMs) are increasingly used for audio deepfake detection (ADD), yet existing benchmarks assess their verdicts or rationale plau
arXiv:2609.38638v1 Announce Type: cross Abstract: Pickup trucks account for 14% of new light-duty vehicles produced in the United States, yet are among the least aerodynamic. Their open cargo bed adds
arXiv:2609.36066v1 Announce Type: cross Abstract: Open-world aerial object-goal search is a foundational yet challenging task, requiring aerial agents to autonomously explore large-scale, unstructured
arXiv:2602.09839v2 Announce Type: replace Abstract: Existing multimodal retrieval benchmarks largely emphasize semantic matching on daily-life images and offer limited diagnostics of professional know
arXiv:2609.36598v1 Announce Type: new Abstract: A video can exhibit convincing motion and photorealism yet fail immediately when visual text collapses. Unlike generic scene content, visual text is unf
arXiv:2609.35879v1 Announce Type: cross Abstract: Benchmarks for large language models (LLMs) typically evaluate the accuracy of answers against fixed reference labels. But a central step in many comp
arXiv:2609.36987v1 Announce Type: new Abstract: Graph databases are increasingly queried through natural language, yet every existing benchmark evaluates isolated single-turn queries rather than the m
arXiv:2609.37871v1 Announce Type: cross Abstract: Average performance on routine driving benchmarks does not establish planner reliability under rare, safety-critical hazards. We proposed ExceptionDri
arXiv:2609.35952v1 Announce Type: cross Abstract: We introduce HEAR (Human-recorded Evaluation of Audio-LLM bias by Real speakers), a large-scale, ecologically valid benchmark comprising 87k real huma
arXiv:2609.37292v1 Announce Type: new Abstract: Physical active vision allows robots to change their viewpoint when task-relevant observations become unreliable, yet existing manipulation benchmarks p
arXiv:2609.36635v1 Announce Type: cross Abstract: Bug validation asks a coding agent to produce an executable witness for a reported bug. The witness combines a concrete input with a testing harness a
arXiv:2609.31675v1 Announce Type: new Abstract: We introduce the Active Causal Discovery Benchmark (ACDB), an SCM-grounded environment for evaluating whether LLM agents recover causal graph structure
arXiv:2607.10427v2 Announce Type: replace Abstract: Local motion blur detection requires pixel-level localization of blurred regions. Existing benchmarks let models rely on gradient shortcuts that fai
arXiv:2609.33980v1 Announce Type: new Abstract: Dynamic graph anomaly detection requires repeated decisions as graph structure and class prevalence drift, yet detector benchmarks usually score a fixed
arXiv:2609.35032v1 Announce Type: cross Abstract: In complex embodied visual reasoning scenarios, an agent often has only a limited field of view, and the evidence needed to answer a question may be d
arXiv:2609.33893v1 Announce Type: new Abstract: Large audio-language models (LALMs) produce fluent responses about audio but often hallucinate by making plausible yet ungrounded claims. Existing audio
arXiv:2609.32810v1 Announce Type: cross Abstract: Multidisciplinary tumor boards integrate multimodal clinical observations and longitudinal patient histories through specialist discussions, yet bench
arXiv:2609.35581v1 Announce Type: cross Abstract: We introduce QC-Stark, a benchmark for evaluating large language models (LLMs) on 11 quantum computing (QC) tasks, spanning circuit construction, debu
arXiv:2609.34384v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models have shown strong capabilities in robotic manipulation, yet existing benchmarks typically assume that task-relevan
arXiv:2609.34390v1 Announce Type: new Abstract: Multi-animal tracking (MAT) supports the study of animal movement, behavior, and group interactions. However, general multi-object tracking (MOT) benchm
arXiv:2609.30298v1 Announce Type: new Abstract: Systematic reviews (SR) are essential for evidence-based research, but their screening phase is highly time-consuming and labor-intensive. Large languag
arXiv:2609.30604v1 Announce Type: cross Abstract: Existing computer-use agent benchmarks do not fully evaluate agents acting as assistants. A useful assistant retrieves information across complex, mul
arXiv:2609.30722v1 Announce Type: cross Abstract: Existing roadside traffic datasets support perception, forecasting, and visual question answering, but they do not evaluate counterfactual video gener
arXiv:2609.30055v1 Announce Type: cross Abstract: In the Era by Eon benchmark, each question states the rules for its answer, and code computes the answer from a generated company's data. When agents
Would be interesting right? Models are starting to score higher and higher on benchmarks like Parametric CAD Bench, but can they pass an actual exam? The Certified SOLIDWORKS Associate (CSWA) exam mig
arXiv:2604.01754v2 Announce Type: replace-cross Abstract: Mathematical reasoning is a hallmark of human intelligence, and whether large language models (LLMs) can meaningfully perform it remains a cen
arXiv:2609.28952v1 Announce Type: new Abstract: Robot-policy benchmarks increasingly cover diverse tasks and preset out-of-distribution conditions, but typically evaluate complete trajectories from pr
arXiv:2609.30027v1 Announce Type: new Abstract: Frontier language models are rarely used in clinical workflows because the realistic, longitudinal benchmarks needed to develop them are scarce. Real el
arXiv:2609.29740v1 Announce Type: cross Abstract: Ligand-based virtual screening (LBVS) is a practical first-pass tool in early-stage drug discovery, but existing benchmarks can overestimate performan
In September 2026, Sebastian Thrun announced **PhilosophyBench**, the first independent, large‑scale benchmark created by Stanford’s AI Lab and HCI group to evaluate artificial intelligence’s philosop
arXiv:2609.27176v1 Announce Type: new Abstract: Evidence that evaluation material entered training does not reveal how much it affected evaluation. This distinction leaves a contaminated benchmark sco
Today, @FactoryAI joins the Specialized Intelligence Index with Legacy-Bench! The SII keeps growing, now with benchmarks testing how models handle the legacy software behind payroll, financial settlem