Knowledge catalogue
💫Very happy to release NeuralBench, to benchmark Neuro AI models and datasets in the open! 🧵Thread, 💻Code, 📝White Paper below: 🧠 Introducing NeuralBench: a unified, open-source framework to benchmark
arXiv:2605.00873v1 Announce Type: cross Abstract: The rapid advancement of photorealistic Text-to-Video (T2V) generation brings in an urgent need for up-to-date evaluation methods. Existing benchmarks
arXiv:2605.01939v1 Announce Type: new Abstract: Static benchmarks for LLMs are increasingly compromised by contamination and overfitting especially on knowledge intensive reasoning tasks While recent
arXiv:2605.01391v1 Announce Type: new Abstract: Existing benchmarks for Vision-Language Models (VLMs) primarily evaluate spatio-temporal understanding on simple single-action videos, closed attribute
arXiv:2605.00674v1 Announce Type: new Abstract: Large language models (LLMs) are becoming increasingly capable mathematical collaborators, but static benchmarks are no longer sufficient for evaluating
arXiv:2604.28177v1 Announce Type: new Abstract: We introduce AEGIS, A holistic benchmark for Evaluating forensic analysis of AI-Generated academic ImageS. Compared to existing benchmarks, AEGIS featur
arXiv:2604.27389v1 Announce Type: cross Abstract: In recent years, Multimodal Large Language Models (MLLMs) have achieved remarkable progress on a wide range of multimodal benchmarks. Despite these ad
arXiv:2601.08611v2 Announce Type: replace-cross Abstract: The growing scale of online misinformation urgently demands Automated Fact-Checking (AFC). Existing benchmarks for evaluating AFC systems, how
arXiv:2508.13024v3 Announce Type: replace Abstract: LLM-based web agents have the potential to automate long-running web tasks, such as searching for products in multiple e-shops and subsequently orde
arXiv:2604.27776v1 Announce Type: new Abstract: While GUI agents have shown impressive capabilities in common computer-use tasks such as OSWorld, current benchmarks mainly focus on isolated and single
arXiv:2604.14858v2 Announce Type: replace Abstract: As agent systems move into increasingly diverse execution settings, trajectory-level safety evaluation and diagnosis require benchmarks that evolve
arXiv:2602.11224v3 Announce Type: replace-cross Abstract: We present Agent-Diff, a novel benchmarking framework for evaluating agentic Large Language Models (LLMs) on real-world productivity software
arXiv:2604.25392v1 Announce Type: new Abstract: This paper benchmarks a classical machine learning approach based on PyCaret AutoML against a deep learning approach based on IndoBERT fine-tuning for b
arXiv:2604.22827v1 Announce Type: new Abstract: Millimeter-wave (mmWave) radar has shown great potential for contactless, privacy-preserving, and robust human sensing, yet existing mmWave-based human
arXiv:2405.10138v2 Announce Type: replace Abstract: In this paper, we introduce the Polish Massive Text Embedding Benchmark (PL-MTEB), a comprehensive benchmark for text embeddings in the Polish langu
arXiv:2604.14709v2 Announce Type: replace Abstract: Existing benchmarks for hardware design primarily evaluate Large Language Models (LLMs) on isolated, component-level tasks such as generating HDL mo
arXiv:2604.21094v1 Announce Type: new Abstract: Graph Neural Networks (GNNs) excel on relational data, but standard benchmarks unrealistically assume the graph is centrally available. In practice, set
GPT-5.5 is likely the best model in the world. But open models like Kimi and Minimax get almost identical coding benchmark scores at 10-25x lower cost. I broke down the benchmarks and pricing. Here's
ParseBench is now live on @Kaggle. The first document OCR benchmark built for AI agents — 2,000 enterprise pages, 167K+ test rules, 5 dimensions that actually break downstream agents. Benchmark your p
arXiv:2508.06226v2 Announce Type: replace Abstract: Geometry problem solving (GPS) poses significant challenges for Multimodal Large Language Models (MLLMs) in diagram comprehension, knowledge applica
arXiv:2604.18487v1 Announce Type: new Abstract: The Adversarial Humanities Benchmark (AHB) evaluates whether model safety refusals survive a shift away from familiar harmful prompt forms. Starting fro
The Reddit post 'Benchmarking programs?' in r/ollama likely discusses tools and methods for measuring the performance of local language models running on Ollama. Available benchmarking tools for Ollam
arXiv:2604.16456v1 Announce Type: new Abstract: Real-time voice assistants must revise task state when users interrupt mid-response, but existing spoken-dialog benchmarks largely evaluate turn-based i
arXiv:2604.17259v1 Announce Type: cross Abstract: User behavior in the real world is diverse, cross-domain, and spans long time horizons. Existing user modeling benchmarks however remain narrow, focus
arXiv:2604.18326v1 Announce Type: new Abstract: Recent advancements in audio-video joint generation models have demonstrated impressive capabilities in content creation. However, generating high-fidel
arXiv:2604.17570v1 Announce Type: new Abstract: Peripheral Blood Smear (PBS) is a critical microscopic examination in hematopathology that yields whole-slide imaging (WSI). Unlike solid tissue patholo
arXiv:2604.17375v1 Announce Type: new Abstract: Recent advances in Vision-Language Models (VLMs) have substantially enhanced their ability across multimodal video understanding benchmarks spanning tem
arXiv:2604.16241v1 Announce Type: cross Abstract: Large language models have shown strong performance on broad-domain knowledge and reasoning benchmarks, but it remains unclear how well language model
arXiv:2604.03611v2 Announce Type: replace Abstract: Portrait composition plays a central role in portrait aesthetics and visual communication, yet existing datasets and benchmarks mainly focus on coar
Agent evals are drifting away from production reality. Most benchmarks use clean tasks, well-specified requirements, deterministic metrics, and retrospective curation. Production work is messier, with
arXiv:2604.13072v1 Announce Type: new Abstract: LLM-based agents are increasingly expected to handle real-world assistant tasks, yet existing benchmarks typically evaluate them under isolated sources
arXiv:2503.21380v3 Announce Type: replace Abstract: The rapid advancement of large reasoning models has saturated existing math benchmarks, underscoring the urgent need for more challenging evaluation
arXiv:2604.09860v1 Announce Type: cross Abstract: The pursuit of general-purpose robotics has yielded impressive foundation models, yet simulation-based benchmarking remains a bottleneck due to rapid
arXiv:2604.09649v1 Announce Type: cross Abstract: Brain-computer interfaces (BCIs) have opened new platforms for human-computer interaction, medical diagnostics, and neurorehabilitation. Wearable BCI
arXiv:2604.09514v1 Announce Type: new Abstract: Recent advances in large language models (LLMs) have enabled the large-scale generation of highly fluent and deceptive news-like content. While prior wo
arXiv:2512.02231v2 Announce Type: replace-cross Abstract: Multimodal large language models (MLLMs) are expected to jointly interpret vision, audio, and language, yet existing video benchmarks rarely a
arXiv:2507.20185v2 Announce Type: replace Abstract: Session history is a common way of recording user interacting behaviors throughout a browsing activity with multiple products. For example, if an us
arXiv:2504.10284v5 Announce Type: replace Abstract: Literature review tables are essential for summarizing and comparing collections of scientific papers. In this paper, we study the automatic generat
arXiv:2604.06185v1 Announce Type: cross Abstract: Fulfilling user needs through Large Language Model multi-turn, multi-step tool-use is rarely a straightforward process. Real user interactions are inh
arXiv:2604.07070v1 Announce Type: new Abstract: While Large Language Models (LLMs) demonstrate remarkable reasoning capabilities, their potential for purpose-driven exploration in dynamic geo-spatial
arXiv:2604.06505v1 Announce Type: cross Abstract: Large language models (LLMs) are widely explored for reasoning-intensive research tasks, yet resources for testing whether they can infer scientific c
AD2-Bench diagnoses multimodal reasoning via a Chain of Evidence, showing failures from spatial ambiguity and semantic uncertainty in adverse urban scenes.
arXiv:2610.22298v1 Announce Type: cross Abstract: Large language models (LLMs) can generate differential diagnoses for complex cases, but a useful second opinion must also be factually grounded, clini
arXiv:2610.21270v1 Announce Type: new Abstract: Trustworthy biomedical AI requires reliable action selection when evidence is missing, stale, conflicting, or late, not only accurate answers on static
arXiv:2512.20159v2 Announce Type: replace-cross Abstract: Code from LLM-assisted development may contain defects like API hallucinations, motivating the development of code evaluation metrics to asses
arXiv:2610.21468v1 Announce Type: cross Abstract: Every story has two orders: the order in which its events happen and the order in which they are told. A story is nonlinear when the two differ. We in
arXiv:2610.21568v1 Announce Type: new Abstract: Recent multimodal large language models have advanced video understanding, but existing benchmarks mainly evaluate whether models can interpret observed
arXiv:2610.20776v1 Announce Type: cross Abstract: LLM-based agents increasingly execute long-horizon, end-to-end workflows involving shell commands, code modification, dependency installation, and int
arXiv:2610.21227v1 Announce Type: new Abstract: We present a Korean Benchmark Suite (KBS) comprising Ko-ARC, Ko-GSM8K, Ko-EQ-Bench, Ko-WinoGrande, Ko-LAMBADA, and Ko-IFEval, covering six capabilities
arXiv:2412.07251v1 Announce Type: cross Abstract: Large language models have exhibited significant enhancements in performance across various tasks. However, the complexity of their evaluation increas
arXiv:2610.19906v1 Announce Type: new Abstract: LLMs have achieved strong performance on chemical prediction benchmarks. However, existing evaluation methods typically provide target entity identity i
arXiv:2610.19118v1 Announce Type: new Abstract: Emerging decision models such as JEV provide typed interfaces that map task instructions and observations directly to choices, judgments, and scores. As
arXiv:2610.19638v1 Announce Type: new Abstract: We present J-tau, the first publicly available Japanese benchmark for evaluating tool-using LLM agents in interactive customer-support scenarios. While
arXiv:2610.14766v1 Announce Type: cross Abstract: AI coding agents now write much of the code that ships, yet whether that code is secure remains unclear. We introduce the Last Secure Code Benchmark,
arXiv:2503.21821v2 Announce Type: replace Abstract: We introduce Physics, a benchmark for evaluating foundation models on advanced physics problem solving. It contains 1034 annotated, open-ended probl
arXiv:2610.16133v1 Announce Type: cross Abstract: Puzzles offer a natural testbed for Large Language Models (LLMs) reasoning because they pose verifiable goals under partial information and require mu
arXiv:2606.14715v2 Announce Type: replace-cross Abstract: LLM agents are increasingly used to simulate real-world interactions, but it remains unclear whether simulated behaviors preserve the content
arXiv:2610.14813v1 Announce Type: new Abstract: Dengue is a viral infection caused by the dengue virus (DENV) and a major public health concern in Bangladesh, where symptom overlap with other febrile
arXiv:2610.14883v1 Announce Type: cross Abstract: We present MARL-BattleGrounds (MARL-BGs), an easy-to-use JAX-native benchmark for heterogeneous and competitive multi-agent reinforcement learning (MA
arXiv:2610.17856v1 Announce Type: new Abstract: Time-series analysis is increasingly studied in multimodal settings, yet most benchmarks pair temporal signals with text or with visualizations derived