Knowledge catalogue
François Chollet observed that all leading contenders on the ARC-AGI-3 benchmark employ a similar methodological approach, suggesting a convergence in techniques for solving abstract reasoning tasks.
DSGym is a comprehensive framework designed to evaluate and train AI agents for data science tasks, providing a structured environment for benchmarking agent performance across various data science wo
arXiv:2606.31004v1 Announce Type: new Abstract: AI-generated video detection benchmarks such as GenVidBench and AIGVDBench are the de facto leaderboards, yet most evaluation protocols leave uncontroll
arXiv:2606.31508v1 Announce Type: new Abstract: Automatic speech recognition for children's reading remains underdeveloped for most African languages, including Bambara, despite its potential value fo
arXiv:2606.31435v1 Announce Type: new Abstract: Data refinement involves executing multi-step recipes over evolving text states, where both composition and execution order of processing operators dete
arXiv:2606.31608v1 Announce Type: new Abstract: Large Language Models (LLMs) achieve strong results on many medical benchmarks, but their clinical reasoning remains difficult to evaluate reliably. A c
arXiv:2603.23455v2 Announce Type: replace Abstract: Multi-Modal LLMs (MLLMs) demonstrate strong visual grounding capabilities on popular object detection benchmarks like OdinW-13 and RefCOCO. However,
DigitalOcean has released Evaluations, a feature for testing production models and routers within their inference stack. This tool enables developers to validate and benchmark their AI/ML models befor
arXiv:2606.30695v1 Announce Type: cross Abstract: Single-cell drug perturbation models should predict not only transcriptional response magnitude, but also whether a treatment alters the proliferative
Multi-GPU kernels are the real test for coding models. Today at @aiDotEngineer, @simran_s_arora shared ParallelKernelBench, an open-source benchmark for evaluating whether LLMs can write fast CUDA ker
arXiv:2606.31895v1 Announce Type: new Abstract: LiDAR has increasingly been integrated into traffic cameras to expand coverage and mitigate occlusion in roadside cooperative perception. However, how u
arXiv:2606.31916v1 Announce Type: new Abstract: Theory of Mind (ToM) benchmarks for Large Language Models (LLMs) typically rely on passive question-answering formats, but the deployment of LLMs in inc
arXiv:2606.31410v1 Announce Type: new Abstract: Graphical user interface (GUI) agents build on vision-language models to complete user tasks end-to-end in real applications through interface actions s
arXiv:2606.29689v1 Announce Type: new Abstract: Open-ended aesthetic critique is a challenge for multimodal large language models (MLLMs): unlike multiple-choice aesthetic benchmarks, it has no single
arXiv:2606.30430v1 Announce Type: cross Abstract: The increasing connectivity of modern vehicles has made securing in-vehicle communication networks a critical challenge. Intrusion Detection Systems (
arXiv:2606.29876v1 Announce Type: cross Abstract: Modern large language models (LLMs) reach 60-70% diagnostic accuracy on complex clinical case benchmarks, but accuracy alone cannot distinguish stable
arXiv:2606.28471v1 Announce Type: new Abstract: Model capability is the central variable in LLM pre-training, yet is never observed directly: data shapes it prospectively, while evaluation reveals it
arXiv:2606.28601v1 Announce Type: cross Abstract: Recent progress in Text-to-SQL has been driven by stronger language models and prompting strategies, yet performance on real enterprise benchmarks suc
arXiv:2606.28551v1 Announce Type: cross Abstract: Building performant Vision-Language Models (VLMs) requires carefully curating large-scale training datasets, yet the community lacks systematic benchm
arXiv:2604.05318v2 Announce Type: replace Abstract: Harmful content detectors, particularly disinformation classifiers, are predominantly developed and evaluated on Standard American English (SAE), le
arXiv:2606.30128v1 Announce Type: new Abstract: Chain-of-thought (CoT) prompting improves LLM reasoning, but the source is contested: do the intermediate steps help because they carry useful semantic
arXiv:2606.29059v1 Announce Type: cross Abstract: World modeling requires forecasting uncertain futures while preserving information useful for downstream perception. Existing visual world models ofte
Fireworks AI announced achieving 446 tokens per second throughput speeds as measured by Artificial Analysis benchmarks, positioning this as their standard performance metric. The company offers custom
arXiv:2606.29915v1 Announce Type: new Abstract: Vision-Language Models (VLMs) often achieve high performance on benchmarks while remaining 'black boxes', yet they remain prone to hallucination or rely
GeneBench-Pro is a comprehensive benchmarking tool or dataset introduced by OpenAI designed to evaluate AI model performance on genomic and gene-related tasks. It likely provides standardized metrics
arXiv:2606.29111v1 Announce Type: new Abstract: When firms deploy autonomous AI, they must decide how much work to leave to the system and how much to keep workers engaged. This decision affects curre
arXiv:2606.29971v1 Announce Type: new Abstract: A growing body of work suggests that the reasoning capabilities of large language models are largely latent in their base form, with post-training prima
arXiv:2606.28433v1 Announce Type: new Abstract: One goal in reinforcement learning (RL) research is to understand general-purpose sequential decision-making, using benchmark simulators as a proxy for
arXiv:2606.28627v1 Announce Type: cross Abstract: The cart-pole swing-up is a canonical benchmark for nonlinear control of underactuated systems, yet an end-to-end guarantee linking the global swing-u
arXiv:2606.28374v1 Announce Type: new Abstract: LLM agents are increasingly improved without weight updates by evolving a natural-language artifact, such as reflections, workflows, playbooks, cheatshe
arXiv:2606.29905v1 Announce Type: new Abstract: Table parsing aims to convert table images into structured, machine-readable representations, a task requiring the joint perception of complex spatial l
arXiv:2511.06090v3 Announce Type: replace-cross Abstract: Optimizing the performance of large-scale software repositories demands expertise in code reasoning and software engineering (SWE) to reduce r
arXiv:2606.28328v1 Announce Type: cross Abstract: In recent years, text clustering has become a critical technique for applications including intent discovery, topic mining, and recommendation systems
arXiv:2606.29933v1 Announce Type: new Abstract: The alignment of language models is typically studied through the lens of capability benchmarks, but the dynamics of how models change during post-train
arXiv:2606.30456v1 Announce Type: cross Abstract: This project investigates whether recent Vision-Language-Action (VLA) models can be transferred from controlled research benchmarks to a real-world ro
arXiv:2606.27500v1 Announce Type: cross Abstract: Large Vision-Language Models (LVLMs) specialized in healthcare are emerging as a promising research direction due to their potential impact in clinica
arXiv:2606.27999v1 Announce Type: new Abstract: Despite the rapid advance of Multimodal Large Language Models (MLLMs) in high-level video understanding, a fundamental bottleneck remains: these models
arXiv:2509.19671v3 Announce Type: replace Abstract: Public datasets of Chest X-Rays (CXRs) have long been a popular benchmark for developing machine learning (ML) computer vision models in healthcare.
The harness can be adapted for other configurations & can also achieve sota performance at lower cost https://ii.inc/blog/post/zenith-research FrontierSWE is probably the hardest practical benchmark o
arXiv:2606.27708v1 Announce Type: new Abstract: Adapting a foundation vision-language encoder to a specialized retrieval task creates a fundamental tradeoff: gains on the target distribution come at t
It is worth being very, very careful about how you are approaching routing, especially when the systems are primarily tested on verifiable IT benchmarks, which may lead you to overestimate the ability
The more serious answer @emollick I actually wouldn't be surprised if we're generally in an AI model release moratorium (regardless of capabilities) until the cybersecurity benchmark is finalized. Not
arXiv:2606.26738v1 Announce Type: new Abstract: While recent advancements in generative image editing models have achieved stunning visual fidelity, it remains an open question whether these systems p
arXiv:2606.26102v1 Announce Type: cross Abstract: Standard post-training pipelines apply supervised fine-tuning (SFT) and reinforcement learning (RL) to make language models helpful, but these process
Nous Research announced an upcoming HermesBench full leaderboard, indicating they are developing or expanding a benchmarking system, likely for evaluating their Hermes model family or related language
arXiv:2606.26346v1 Announce Type: new Abstract: Agentic benchmarks have emerged across general-purpose and domain-specific settings, including finance, coding, law, and drug discovery, yet energy-doma
Nous Research is conducting benchmarking tests to evaluate whether combinations of open-source models can achieve performance comparable to Anthropic's Claude Opus while maintaining significantly lowe
arXiv:2606.27369v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) for training LLMs typically rely on ground-truth answers to assign rewards, limiting their applica
arXiv:2606.14397v2 Announce Type: replace Abstract: As agentic systems continue to evolve and are widely deployed in real-world scenarios, there is a growing demand to faithfully evaluate their capabi
arXiv:2606.26947v1 Announce Type: cross Abstract: While personalized image generation has achieved remarkable progress, multi-reference image generation (MRIG) remains a challenging task. Most existin
arXiv:2606.26525v1 Announce Type: new Abstract: Auto-formalization is critical for scalable formal verification, but existing progress largely focuses on isolated statements, while theory-scale auto-f
Meta released Harness, an open-source framework for evaluating and benchmarking AI model performance across diverse tasks and datasets. The tool enables standardized testing of language models and aim
arXiv:2602.11933v2 Announce Type: replace Abstract: End-to-End Speech Translation (E2E-ST) has seen significant advancements, yet current models are primarily benchmarked on curated, 'clean' datasets.
arXiv:2606.25454v1 Announce Type: new Abstract: In this study, the learning-based inertial odometry problem is investigated using raw IMU measurements obtained from the EuRoC MAV benchmark dataset. In
arXiv:2606.25013v1 Announce Type: cross Abstract: Today's reasoning models use thinking tokens to attain stronger performance on benchmarks than their instruction-tuned counterparts. It is also genera
arXiv:2606.17412v3 Announce Type: replace Abstract: Pathological images are inherently multi-scale, requiring pathologists to integrate evidence from global tissue architecture at low magnification to
Explore how the GitHub Copilot agentic harness delivers strong results across multiple benchmarks and leading token efficiency, while maintaining flexibility to choose among more than 20 models. The p
In this episode, @OpenAI Chief Research Officer @markchen90 joins @allenpark to flambé shrimp, cook Korean stew, and chat about being at the frontier of AI research: why scaling laws and pre-training
LLMs are getting better at writing GPU kernels. Multi-GPU kernels are the harder test. At @aiDotEngineer World's Fair, @simran_s_arora will share ParallelKernelBench, an open-source benchmark built fr
Cursor discusses methods for improving AI evaluation environments to prevent reward hacking and ensure benchmark scores more accurately measure model capabilities rather than exploitation of evaluatio