Knowledge catalogue
arXiv:2609.26945v1 Announce Type: new Abstract: Judicial reasoning remains challenging for large language models (LLMs) to analyze. This paper contributes a sentence-level benchmark for evaluating the
Callum Borchers / Wall Street Journal: Companies are increasingly using AI benchmarking services to aggregate public job listings and other payroll data to identify under- and over-paid employees — Ne
Ok, so the Mac M5 Ultra (256GB) hit the market, but the only publicly available benchmarks material are flashy YouTube 'clown influencers' videos. We need serious numbers to evaluate whether Apple’s s
arXiv:2609.26942v1 Announce Type: cross Abstract: Current literature evaluates large language models (LLMs) on multilingual kinship understanding using multiple choice benchmarks, treating it as a rec
arXiv:2508.13680v5 Announce Type: replace Abstract: We introduce VMMU, a Vietnamese Multitask Multimodal Understanding and Reasoning Benchmark designed to evaluate how vision-language models (VLMs) in
arXiv:2609.22696v1 Announce Type: new Abstract: Decentralized social media platforms create new opportunities and challenges for computational mental health research because data access, moderation, l
arXiv:2609.22529v1 Announce Type: new Abstract: International law provides the normative framework through which states coordinate action, regulate armed conflict, and protect human rights, yet its te
arXiv:2609.25813v1 Announce Type: new Abstract: This short report documents our entry to the COMFORT Localization Benchmark (IROS 2026), evaluated on the GrandTour dataset recorded with the Boxi paylo
Most mental health benchmarks focus on emergency situations. MentalHealthBench is designed to cover the full spectrum of mental health conversations that people bring to AI - from everyday support to
arXiv:2609.24099v1 Announce Type: new Abstract: Body-cue recognition can support assistive robots, but benchmark accuracy does not guarantee reliable behavior under a robot-camera viewpoint. We presen
arXiv:2609.23231v1 Announce Type: new Abstract: Cross-lingual information retrieval (CLIR) is increasingly important in multi-national industries, where critical technical evidence may exist in a diff
arXiv:2609.23825v1 Announce Type: new Abstract: We present a comprehensive benchmark of Federated Learning (FL) for multilingual Automatic Speech Recognition (ASR), evaluating four Speech-LLM architec
arXiv:2609.22308v1 Announce Type: new Abstract: Coding-agent benchmarks usually evaluate implementation after the target behavior has been specified in text, code, or demonstrations. Existing research
arXiv:2609.22092v1 Announce Type: cross Abstract: Electroencephalography (EEG) is a low-cost and non-invasive signal source for dementia screening, yet existing EEG-based studies remain difficult to c
arXiv:2505.12254v3 Announce Type: replace Abstract: Existing visual place recognition (VPR) datasets predominantly rely on vehicle-mounted imagery, offer limited multimodal diversity, and underreprese
arXiv:2609.22200v1 Announce Type: new Abstract: LLM assistants and agentic systems log long multi-turn conversations. AI providers often scan these conversations for Personally Identifiable Informatio
arXiv:2609.23427v1 Announce Type: new Abstract: Vision foundation models targeting Earth observation (EO) tasks are commonly evaluated on clean downstream benchmarks, but operational EO products can a
arXiv:2509.01167v3 Announce Type: replace-cross Abstract: Vision-language models (VLMs) can ingest only a limited number of video frames, making frame selection a practical necessity. But do current V
arXiv:2609.21763v1 Announce Type: new Abstract: A medical model's benchmark score does not establish that the same conclusion holds under a different evaluation. This study tests whether claims about
arXiv:2609.22064v1 Announce Type: new Abstract: Advances in large-scale neural recording have made it possible to collect data across many animals and distributed brain regions, raising the question o
arXiv:2609.21344v1 Announce Type: cross Abstract: For Internet of Things (IoT) devices, a secure algorithm alone is not enough: an attacker with physical access can attack the implementation directly,
arXiv:2609.21493v1 Announce Type: new Abstract: Multimodal large language models, or MLLMs, perform well at visual understanding and structured generation, yet these capabilities do not establish whet
arXiv:2609.21371v1 Announce Type: new Abstract: Social Proactive Intelligence (SPI) extends proactive assistance beyond task completeness to consider social appropriateness in diverse embodied scenari
Thanks for the model, we were able to port Laya to coreml with 99.5% of the ops on ANE + benchmarked too. it is now blazing fast with 3.7 ms per decision on an M5 Pro. Release: https://github.com/Flui
In February OpenAI stopped reporting SWE-bench Verified and recommended other labs stop too. Every frontier model they tested could reproduce the human-written reference fix, or verbatim details of th
NVIDIA AIPerf is a benchmarking framework that supersedes GenAI‑Perf with a multiprocess architecture, eliminating client‑side bottlenecks during high‑concurrency LLM testing. It supports more than 15
arXiv:2609.19879v1 Announce Type: new Abstract: Question answering has advanced rapidly with large language models, but predominantly for high-resource languages, in both text and spoken settings. Spo
arXiv:2609.17910v1 Announce Type: new Abstract: We introduce Map2Route, a human-curated benchmark for compositional language-grounded route planning over pre-built semantic maps. Map2Route contains 1,
Julie Bort / TechCrunch: PrismML releases Bonsai 2 27B, which compresses Alibaba's Qwen3.8 27B to 5.9 GB, small enough for smartphones, while retaining 98.2% of Qwen's benchmark scores — If AI lab Pri
arXiv:2609.18156v1 Announce Type: new Abstract: Teochew has a substantial speaker community and exhibits distinctive lexical, syntactic, and pragmatic features, yet textual resources for evaluating la
arXiv:2609.16075v1 Announce Type: cross Abstract: Flexible robotic production requires joint decisions on process progression, material routing, resource assignment, temporary cooperation, and simulta
MLPerf Inference v6.1 introduced the first agentic workload run on datacenter hardware and evaluated a model with more than one trillion parameters (Kimi K2), setting a new benchmark for scale. The te
the independence of AI evaluators is going to matter more... so i made it a 'benchmark': http://evaluatorbench.com (research preview) source-linked directory of third-party evaluators of frontier AI.
arXiv:2609.14302v1 Announce Type: new Abstract: Can financial vision-language models (VLMs) turn chart evidence into reliable action recommendations? Existing hallucination evaluations are mostly clai
ELON MUSK: I saw some pretty funny jokes on 𝕏. One of them I saw was: 'Your girlfriend's a 10, but she's a benchmark maxxer.' 🤣 Media All-In Summit: Elon Musk & Gwynne Shotwell -- AI Risks and Peer Re
arXiv:2609.15625v1 Announce Type: cross Abstract: Driven by the escalating demand for privacy-preserving computing, Federated Learning (FL) has witnessed remarkable progress, becoming a cornerstone te
arXiv:2609.13637v1 Announce Type: new Abstract: Persistent agents need evaluations that distinguish identity facts they can recall from those they express and enact. We introduce PAI-Bench, a provider
arXiv:2609.14495v1 Announce Type: new Abstract: Image colorization is an inherently ill-posed task, since a single grayscale image may correspond to multiple plausible colorized results. Consequently,
arXiv:2609.13659v1 Announce Type: cross Abstract: Underwater Acoustic Target Recognition (UATR) of ships is well-suited for machine learning, yet its progress is hindered by the lack of large, diverse
arXiv:2609.12509v1 Announce Type: new Abstract: Evaluation of vision-language models (VLMs) for multimodal mathematical reasoning remains limited for low-resource languages and for geometry problems t
deepseek v4.1 flash has arrived on together ai it beats gpt-5.6 sol on agentic benchmarks at one third the cost per task the model supports a 1m context window and native multimodal input, with 552b t
arXiv:2609.11141v1 Announce Type: new Abstract: Large Language Models (LLMs) are increasingly used to generate structured outputs, but their reliability remains unclear when those outputs must satisfy
Fugu Ultra v2 by the numbers. Full Results: https://sakana.ai/fugu-max-release/ • #1 on 5 out of 8 hard benchmarks (including DeepSWE, Chartography, Toolathon) • DeepSWE: 74.3 • Chartography: 48.3 vs
arXiv:2609.10099v1 Announce Type: new Abstract: Molecule generation has emerged as a powerful computational tool for de novo drug design, enabling the exploration of chemical space beyond the limits o
arXiv:2609.05818v1 Announce Type: new Abstract: We introduce ABLE, a benchmark for evaluating LLM agents' ability to use biological AI models (BAIMs), such as ProteinMPNN and AlphaFold3, in dual-use p
Another goated release by DeepSeek. Open weight, btw. Outperforms Opus 5 on key benchmarks, btw. Lower prices too. Looks like an extremely efficient model. What a legendary run. 🚀 Introducing DeepSeek
arXiv:2609.10243v1 Announce Type: new Abstract: Embodied AI, including vision-language-action and world-action models, must operate reliably in the physical world. Yet methods that perform well in sim
arXiv:2609.10539v1 Announce Type: new Abstract: A research idea may be novel, coherent, and scientifically plausible, yet its proposed method may remain insufficiently specified for faithful implement
arXiv:2609.09482v1 Announce Type: new Abstract: Critical infrastructure location data is often incomplete and unevenly distributed globally, especially in developing regions. Earth observation foundat
arXiv:2609.05760v1 Announce Type: cross Abstract: We present RAGMark, a modular benchmarking framework for advanced Retrieval-Augmented Generation (RAG) systems targeting small-scale multi-GPU environ
A language model walks into a benchmark. Scores 99.9% on MMLU. Wins a gold medal at the IMO. Solves SWE-bench. Passes Humanity’s Last Exam. Resists every jailbreak from Pliny the Liberator. The first
arXiv:2609.05435v1 Announce Type: new Abstract: Modern language agents are expected to operate over long horizons: they ask follow-up questions, reuse worked examples, handle tool feedback, and adapt
arXiv:2609.07600v1 Announce Type: new Abstract: Tree species recognition supports forest inventory and biodiversity monitoring but still depends on scarce taxonomic expertise. Bark is visible year-rou
arXiv:2609.06353v1 Announce Type: new Abstract: Understanding collaborative behavior in children is important for analyzing social participation, peer interaction, shared attention, and engagement dur
arXiv:2609.05518v1 Announce Type: new Abstract: Despite the strong capabilities of multimodal large language models (MLLMs), their parametric knowledge remains incomplete and difficult to update, moti
arXiv:2609.08117v1 Announce Type: new Abstract: Driver motion can provide cues to ongoing behavior, attention, and near-term driving intent. However, most existing driver-centric datasets focus on rec
arXiv:2609.08209v1 Announce Type: new Abstract: Learning from Observation (LfO) is a fundamental robotic capability that replicates how humans and animals socially learn from each other. Beyond its bi
arXiv:2609.08040v1 Announce Type: cross Abstract: The software supply chain has become an increasingly exposed attack surface because of its reliance on intricate yet fragile dependencies. Existing de
We benchmarked DeepSeek V4.1 Flash by @deepseek_ai . It reached 98% of GPT-6 Astra’s score at 1.4% of the cost on everyday design tasks based on user requests. Every model except Astra scored lower AN
arXiv:2609.06147v1 Announce Type: cross Abstract: Ask a language model the same question about the same document twenty times, and it sometimes returns two different answers. We built Probity, a bench