Search: “benchmark”

Fugu stands shoulder-to-shoulder with leading models like Fable and Mythos across the industry's most rigorous engineering, scientific, and reasoning benchmarks. Read the full blog: https://sakana.ai/

Today on AI:AM: @swyx, who coined 'AI Engineer', is gearing up for his next @aidotengineer World's Fair next week We'll be talking about his new FrontierCode benchmark, the state of SWE, whether AI co

Fireworks AI reports that GLM 5.2, their language model, demonstrates performance comparable to GPT 5.5 across various evaluation benchmarks and real-world use cases. This claim suggests GLM 5.2 is co

arXiv:2606.11783v1 Announce Type: new Abstract: Recent progress in video generation has shown impressive visual synthesis capabilities. However, open-domain customized video generation remains limited

arXiv:2606.11464v1 Announce Type: new Abstract: Robotic table tennis is a representative benchmark for high-speed, closed-loop robotic control in dynamic environments, where accurate and fast predicti

arXiv:2606.11976v1 Announce Type: cross Abstract: Software engineering tools increasingly rely on LLM based agents to localize files to change to resolve a software issue. Most AI agents explore repos

arXiv:2510.23320v2 Announce Type: replace-cross Abstract: We introduce LibriConvo, a synthetic conversational speech corpus for speaker diarization and automatic speech recognition (ASR), built by ins

arXiv:2603.00461v3 Announce Type: replace Abstract: We present ReMoT, a unified training paradigm to systematically address the fundamental shortcomings of VLMs in spatio-temporal consistency -- a cri

arXiv:2606.11543v1 Announce Type: new Abstract: Agent Skills augment large language model (LLM) agents with procedural knowledge at inference time, but current benchmarks rarely distinguish what a Ski

arXiv:2606.10381v1 Announce Type: cross Abstract: Muon collider research spans accelerator physics, detector instrumentation, and high-energy phenomenology, with relevant evidence scattered across a r

arXiv:2606.10657v1 Announce Type: new Abstract: Multiple-choice (MCQA) benchmarks are the standard for evaluating pretrained large language models, but their reliance on log-likelihood scoring makes t

arXiv:2606.09874v1 Announce Type: new Abstract: Reconstruction-based methods are widely used for time series anomaly detection, where models are trained to reconstruct subsequences, and anomalies are

arXiv:2606.10778v1 Announce Type: new Abstract: Foundation Models (FMs) have recently redefined the state-of-the-art in histopathology by providing robust representations for whole-slide image (WSI) a

arXiv:2606.10933v1 Announce Type: new Abstract: LLM-based coding agents are usually evaluated in familiar software settings: mainstream languages, common libraries, and public repositories. These benc

arXiv:2602.12966v2 Announce Type: replace Abstract: Understanding how and why large language models (LLMs) fail is becoming a central challenge as models rapidly evolve and static evaluations fall beh

arXiv:2606.09873v1 Announce Type: cross Abstract: Reasoning models achieve strong performance on challenging tasks by generating explicit intermediate reasoning traces before producing a final answer.

Try it out in Comfy today! 🎥Bria’s V-RMBG 3.0 is live! A state-of-the-art upgrade to the only end-to-end video background removal suite in production today. Benchmarked against every major competitor.

arXiv:2606.11079v1 Announce Type: new Abstract: Evaluation remains a critical bottleneck for interactive agent development. Existing evaluation methods often rely on static benchmarks, which fail to c

arXiv:2603.26763v2 Announce Type: replace Abstract: Talking-head videos constitute a predominant content type in real-time communication, yet publicly available datasets for video processing research

NVIDIA FLARE Auto-FL automates federated learning research by constraining agent actions through a control plane, enforcing fixed benchmark contracts, and using an experiment ledger to ensure reproduc

Anthropic PBC today introduced Claude Mythos 5 and Claude Fable 5, two large language models that it says outperform the competition across a wide range of benchmarks. The LLMs are derived from the Cl

arXiv:2606.08840v1 Announce Type: new Abstract: Code generation models are typically compared using compact execution benchmarks and aggregate pass rates, but such summaries obscure how performance va

BREAKING: Anthropic just dropped Claude Fable 5—this is Mythos, made safe for public release. It is the best coding model in the world. We've been testing it internally @every for the last week or so

arXiv:2606.08959v1 Announce Type: new Abstract: We introduce ChinaHeritaQA, a multimodal benchmark dataset for evaluating the cultural reasoning abilities of vision-language models (VLMs) on UNESCO Wo

arXiv:2606.08168v1 Announce Type: cross Abstract: Leading commercial endpoint detection and response (EDR) products have shifted from operator-configured rule sets to multi-component systems where aut

arXiv:2602.17911v3 Announce Type: replace-cross Abstract: Current biomedical question answering (QA) systems often assume that medical knowledge applies uniformly, yet real-world clinical reasoning is

arXiv:2606.09243v1 Announce Type: cross Abstract: Estimating full-hand grasp pressure from egocentric video is critical for immersive VR and robotic manipulation, yet dense tactile sensing often relie

arXiv:2606.09314v1 Announce Type: new Abstract: Generating high-quality dexterous grasps remains challenging for learning-based methods, which often depend on carefully tuned contact losses or costly

LiteParse is a parsing tool that demonstrates exceptionally fast performance, reportedly so rapid that Claude 3.5 Sonnet struggles to believe the benchmarked results are genuine. The post highlights L

arXiv:2606.09748v1 Announce Type: new Abstract: Existing benchmarks for deep research agents (DRAs) assess only single-shot outputs, ignoring a key question: can DRAs improve their reports when guided

arXiv:2606.09390v1 Announce Type: cross Abstract: Body movement communicates intent at distances and in conditions where neither the face, nor speech can be captured. We study the recognition of commu

arXiv:2606.09303v1 Announce Type: new Abstract: The rapid development of pretrained foundation models has enabled more general image segmentation. Multimodal large language models (MLLMs) have been wi

arXiv:2606.08376v1 Announce Type: cross Abstract: As artificial intelligence (AI) systems are increasingly deployed across socially consequential domains, reports of AI-related harms and failures have

Claude Fable 5 is compared against other AI models on various evaluation metrics through Cursor's benchmarking tool. The evaluation likely covers performance across different tasks such as coding, rea

arXiv:2606.08278v1 Announce Type: new Abstract: Humanoid foundation models are advancing faster than we can evaluate them. While real-world testing is expensive and difficult to reproduce, existing si

arXiv:2606.07682v1 Announce Type: cross Abstract: AI agents are increasingly expected to complete long-horizon workflows that require sustained progress over hours, millions of tokens, and complex env

arXiv:2606.06666v1 Announce Type: new Abstract: Deepfake detection systems achieve near-perfect accuracy on benchmarks, yet forensic deployment demands reliable prediction uncertainty. Existing uncert

arXiv:2606.06546v1 Announce Type: new Abstract: Evaluating large language models (LLMs) for education requires measuring how models teach, not only what they know. Existing benchmarks emphasize domain

arXiv:2603.13428v2 Announce Type: replace-cross Abstract: With AI agents increasingly deployed as long-running systems, it becomes essential to autonomously construct and continuously evolve customize

Great tips. In practice, this is how it roughly looks to run agents autonomously for hours or days. /goal or /loop to keep it going. Verification is crucial here. Seeing a number of benchmarks showing

arXiv:2606.07515v1 Announce Type: cross Abstract: We investigate the probabilistic reasoning capabilities of large language models through a controlled benchmarking study on discrete probability probl

arXiv:2606.07368v1 Announce Type: cross Abstract: Automated mitosis detection is a well-established task in computational pathology. While previous benchmarks focused on scanner-induced domain shift,

arXiv:2606.06941v1 Announce Type: new Abstract: Large language models (LLMs) now solve a wide range of expert-level exams at or above human level, yet remain brittle on specialised, evidence-intensive

arXiv:2606.06837v1 Announce Type: cross Abstract: Scripted vs spontaneous speech detection is appealing for interview guardrails, but benchmark performance can be inflated by shortcuts tied to corpus

arXiv:2606.06926v1 Announce Type: new Abstract: While highlight detection for long-form videos is of great practical importance, most existing methods remain limited to short-form content, largely due

arXiv:2606.07435v1 Announce Type: cross Abstract: Visual speech recognition (VSR) models now surpass human lipreaders on benchmarks, but do such gains establish human-like visual speech perception? To