Search: “benchmark”

arXiv:2608.03219v1 Announce Type: new Abstract: Benchmark gains are often treated as evidence of greater LLM capability. Yet the same gain can reflect different changes in model behavior. A model may

Skill libraries are shipping in agent harnesses on the assumption that writing skills down compounds. A new benchmark tests that directly. ContinualSkillBench covers five domains, each with 100 interc

arXiv:2608.03095v1 Announce Type: new Abstract: We present VIVID (Vietnamese Idioms for Validation and Interpretation Depth), the first systematic benchmark for evaluating culturally grounded figurati

arXiv:2608.03499v1 Announce Type: new Abstract: Recent advances in persistent personal-agent frameworks are making human-centered agent networks realistic deployment targets: each user can be served b

arXiv:2608.01291v1 Announce Type: new Abstract: We present ArabicDialectSafety, a human-curated Arabic safety dataset of 25,071 prompts covering six Arabic varieties: Modern Standard Arabic, Syrian, E

arXiv:2608.00652v1 Announce Type: new Abstract: Human assistance in robotics spans around several tasks such as navigation, object manipulation, and placement, where a key challenge is selecting targe

arXiv:2608.01948v1 Announce Type: new Abstract: Long-horizon event-based action understanding remains underexplored because existing datasets largely comprise short, trimmed clips, while collecting na

arXiv:2608.02315v1 Announce Type: new Abstract: Geospatial foundation models aim to learn representations that transfer across regions and sensors, yet evaluating them on specific tasks requires large

arXiv:2608.00736v1 Announce Type: new Abstract: Restoring severely degraded visual media still remains a formidable challenge, as existing methods often hallucinate unnatural textures and contents, st

arXiv:2608.01347v1 Announce Type: new Abstract: Large reasoning models used as coding agents incur costs from deliberation, tool calls, and repeated agent turns, yet the causal effect of prompt wordin

arXiv:2607.29677v1 Announce Type: new Abstract: Enterprise workflows increasingly rely on agents for schema-guided extraction: given a document and a user-defined schema, the agent faithfully follows

Of course, the concern is that in the end, this thread will be fed into the models' training data, but I feel benchmarking isn't so open and very fragmented. submitted by /u/jinnyjuice [link] [comment

Nice benchmark to measure agentic e-commerce capabilities. They ran an agent for one simulated year of e-commerce operations and it ends up with 27.3% of the money a human makes. MerchantBench is a 36

Luz Ding / Bloomberg: Alibaba says its 2.4T-parameter Qwen3.8-Max tops Moonshot's Kimi K3 on some benchmarks, and it plans to release Qwen3.8-Max and Qwen3.8-27B's weights next week — Alibaba Group Ho

Thanks to the community help I finally launched this llm. LM Studio refused to load weight onto second GPU but Unsloth Studio did so everything was done in there. Not a proper benchmark (used PC in pa

arXiv:2607.27405v1 Announce Type: new Abstract: Both expressions of uncertainty and inferences are ubiquitous in natural language, and valid inferences over natural-language expressions of uncertainty

Newley Purnell / Bloomberg: DeepSeek rolls out the official V4 Flash API in public beta, touting enhanced agent capabilities and benchmark scores “far surpassing” V4 Pro Preview — China's DeepSeek rol

🚀 DeepSeek-V4-Flash Official API is now LIVE in public beta! 🔷 We’ve massively upgraded its Agent capabilities—benchmark scores are now far surpassing the V4-Pro-Preview. Check out the massive perform

arXiv:2607.27263v1 Announce Type: new Abstract: Most benchmarks for causal inference over time series are observational, small, or domain-specific, leaving interventional and counterfactual estimation

arXiv:2607.27616v1 Announce Type: new Abstract: Text-to-image and personalized editing models now synthesize high-fidelity single-subject images with ease. Yet placing multiple named people into share

arXiv:2607.26455v1 Announce Type: new Abstract: Large language models (LLMs) have demonstrated strong capabilities in knowledge acquisition and reasoning, yet their ability to retain previously acquir

arXiv:2607.27084v1 Announce Type: new Abstract: Scientific images are the core elements of presenting experimental conclusions, elaborating system architecture, and supporting comparative arguments in

A benchmark score reflects the model as well as the harness and settings used to run it. For long-running agents, retaining reasoning and compacting context lets the model build on what it has already

arXiv:2607.25518v1 Announce Type: new Abstract: Computational AMP discovery is often evaluated through AMP/non-AMP recognition, yet follow-up decisions depend on assay-derived evidence such as target-

BREAKING: Grok 4.5 (high) ranks #1 on the HighWalk benchmark, which tests how well AI agents update technical specifications from code changes. Grok delivered the best combination of quality and opera

BREAKING: Grok 4.5 just claimed the top spot on the new HighWalk Benchmark. The independent test measures how well AI models update real technical specifications from 46 Laravel commits — heavy on cod

arXiv:2607.25589v1 Announce Type: cross Abstract: Medical-imaging AI benchmarks combine datasets, DICOM rendering, prompts, provider APIs, automated labels, statistical code, manuscripts, and reposito

GPT-5.6 Sol has been used to solve open problems in mathematics. So why was it struggling with ARC-AGI-3, a benchmark of 2D puzzle games? We investigated. The harness was not letting it remember what

arXiv:2607.25375v1 Announce Type: new Abstract: India is a vast nation of over 1.4 billion people, varied by hundreds of diverse and locally specific traditions and cultures and 22 officially recogniz

arXiv:2607.25232v1 Announce Type: new Abstract: Digital phenotyping (DP) using smartphones and wearable devices has shown considerable potential for mental health monitoring. However, progress remains

arXiv:2606.13040v2 Announce Type: replace Abstract: Vision-language models (VLMs) are increasingly explored as visual critics, reward generators, and failure detectors in robotic manipulation. These r

arXiv:2607.24810v1 Announce Type: new Abstract: Vision-language models (VLMs) have achieved strong performance on general remote sensing tasks. However, their capability for rare scenes remains insuff

arXiv:2607.25886v1 Announce Type: cross Abstract: Recursive self-improvement requires turning evidence of model failures into better models. Data-centric post-training research entails diagnosing capa

arXiv:2607.22759v1 Announce Type: cross Abstract: Large language models (LLMs) show promise in code generation, but their capabilities to produce correct, synthesizable hardware description language (

arXiv:2607.23803v1 Announce Type: new Abstract: Multi-view Multi-object Association and Tracking (MvMoAT) associates objects across camera views and tracks them over time, supporting identity persiste

arXiv:2607.22635v1 Announce Type: new Abstract: Target-oriented dialogue systems have demonstrated strong capabilities in completing user goals through interactive conversations. However, existing stu

arXiv:2607.22705v1 Announce Type: new Abstract: Object-centric learning aims to represent scenes as objects whose properties can be reused in new combinations. Existing evaluations usually score segme

arXiv:2607.23123v1 Announce Type: new Abstract: Existing evaluations of large language models cover knowledge, reasoning, coding, and tool use, but they rarely treat a verifiable deliverable produced

arXiv:2607.22067v1 Announce Type: new Abstract: The integration of large language models (LLMs) into the nuclear power industry requires outputs grounded in domain-specific knowledge. This study evalu

arXiv:2607.22139v1 Announce Type: new Abstract: Accurate pixel-level classification of coronary angiograms is critical for cardiovascular disease assessment, yet the field lacks standardized evaluatio

On FrontierCode 1.1 Extended, our benchmark for real-world engineering tasks that grades mergeability and quality, Kimi K3 scores 58.2% with a 63.6% pass rate. Within Devin, it excels on reproducing b

One of the least explored questions in AI is how to build rich open-ended environments for learning. Benchmarks evaluate intelligence, environments shape it. iLands is interesting because it doesn't j

Really interesting concept -- instead of optimizing against static benchmarks or internally generated reward models, agents are evaluated through real economic interactions. Using an external market a

arXiv:2607.21866v1 Announce Type: new Abstract: Prior classical-ML learning-curve work fits power laws to tree, linear, and kernel models on tabular data, but at small scale: typically one curve, one

training to benchmarks ≠ getting to AGI This suggests that Opus 5 's gain on ARC-AGI-3 was the result of specific training to improve on that eval, and not a generalized increase in abstract reasoning

An interesting thing I'm observing from the blog/system card is that on a good chunk of the reported benchmarks (~20-30% from a skim), Opus 5 max thinking leads to a degradation in performance compare