Search: “benchmark”

Revisiting Performance Claims for Chest X-Ray Models Using Clinical Context

arXiv:2509.19671v3 Announce Type: replace Abstract: Public datasets of Chest X-Rays (CXRs) have long been a popular benchmark for developing machine learning (ML) computer vision models in healthcare.

29 Jun 2026

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

The harness can be adapted for other configurations & can also achieve sota performance at lower cost https://ii.inc/blog/post/zenith-resear…

DGX agent

The harness can be adapted for other configurations & can also achieve sota performance at lower cost https://ii.inc/blog/post/zenith-research FrontierSWE is probably the hardest practical benchmark o

model-releasesemad-mostaque--x

29 Jun 2026

ZooClaw-FashionSigLIP2: Distilled Fine-tuning for Robust Fashion Retrieval

arXiv:2606.27708v1 Announce Type: new Abstract: Adapting a foundation vision-language encoder to a specialized retrieval task creates a fundamental tradeoff: gains on the target distribution come at t

29 Jun 2026

Applications

It is worth being very, very careful about how you are approaching routing, especially when the systems are primarily tested on verifiable I…

applicationsethan-mollick--x

It is worth being very, very careful about how you are approaching routing, especially when the systems are primarily tested on verifiable IT benchmarks, which may lead you to overestimate the ability

28 Jun 2026

The more serious answer

model-releasesethan-mollick--x

The more serious answer @emollick I actually wouldn't be surprised if we're generally in an AI model release moratorium (regardless of capabilities) until the cybersecurity benchmark is finalized. Not

28 Jun 2026

Do Image Editing Models Understand Lighting?

arXiv:2606.26738v1 Announce Type: new Abstract: While recent advancements in generative image editing models have achieved stunning visual fidelity, it remains an open question whether these systems p

Helpfulness Hurts: Domain-Dependent Degradation of Mid-Trained Compassion Values Under Post-Training

arXiv:2606.26102v1 Announce Type: cross Abstract: Standard post-training pipelines apply supervised fine-tuning (SFT) and reinforcement learning (RL) to make language models helpful, but these process

HermesBench full leaderboard coming soon. Stay tuned!

Nous Research announced an upcoming HermesBench full leaderboard, indicating they are developing or expanding a benchmarking system, likely for evaluating their Hermes model family or related language

researchnous-research--x

How Do Tool-Augmented LLM Agents Perform on Real-World Energy Analytics Tasks?

arXiv:2606.26346v1 Announce Type: new Abstract: Agentic benchmarks have emerged across general-purpose and domain-specific settings, including finance, coding, law, and drug discovery, yet energy-doma

@NousResearch We are working on benching various combos of open source models to see if we can get Opus levels with much cheaper models as w…

Nous Research is conducting benchmarking tests to evaluate whether combinations of open-source models can achieve performance comparable to Anthropic's Claude Opus while maintaining significantly lowe

researchnous-research--x

Reinforcement Learning without Ground-Truth Solutions can Improve LLMs

arXiv:2606.27369v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) for training LLMs typically rely on ground-truth answers to assign rewards, limiting their applica

Running the Gauntlet: Re-evaluating the Capabilities of Agents Beyond Familiar Environments

arXiv:2606.14397v2 Announce Type: replace Abstract: As agentic systems continue to evolve and are widely deployed in real-world scenarios, there is a growing demand to faithfully evaluate their capabi

Scaling Multi-Reference Image Generation with Dynamic Reward Optimization

arXiv:2606.26947v1 Announce Type: cross Abstract: While personalized image generation has achieved remarkable progress, multi-reference image generation (MRIG) remains a challenging task. Most existin

Theory-Scale Auto-Formalization of Logics for Computer Science

arXiv:2606.26525v1 Announce Type: new Abstract: Auto-formalization is critical for scalable formal verification, but existing progress largely focuses on isolated statements, while theory-scale auto-f

Tools

[AINews] It's Meta-Harness Summer

Meta released Harness, an open-source framework for evaluating and benchmarking AI model performance across diverse tasks and datasets. The tool enables standardized testing of language models and aim

toolslatent-space

Applications

Cross-Modal Robustness Transfer (CMRT): Training Robust Speech Translation Models Using Adversarial Text

arXiv:2602.11933v2 Announce Type: replace Abstract: End-to-End Speech Translation (E2E-ST) has seen significant advancements, yet current models are primarily benchmarked on curated, 'clean' datasets.

applicationsarxiv-cs-cl

Delta-Position Estimation-Based IMU Odometry: A Comparison of MLP and Kolmogorov-Arnold Networks

model-releasesarxiv-cs-ro

arXiv:2606.25454v1 Announce Type: new Abstract: In this study, the learning-based inertial odometry problem is investigated using raw IMU measurements obtained from the EuRoC MAV benchmark dataset. In

Do Thinking Tokens Help with Safety?

model-releasesarxiv-cs-cl

arXiv:2606.25013v1 Announce Type: cross Abstract: Today's reasoning models use thinking tokens to attain stronger performance on benchmarks than their instruction-tuned counterparts. It is also genera

Enhancing Pathological VLMs with Cross-scale Reasoning

arXiv:2606.17412v3 Announce Type: replace Abstract: Pathological images are inherently multi-scale, requiring pathologists to integrate evidence from global tissue architecture at low magnification to

Agents

Evaluating performance and efficiency of the GitHub Copilot agentic harness across models and tasks

Explore how the GitHub Copilot agentic harness delivers strong results across multiple benchmarks and leading token efficiency, while maintaining flexibility to choose among more than 20 models. The p

agentsgithub-ai-blog

In this episode, @OpenAI Chief Research Officer @markchen90 joins @allenpark to flambé shrimp, cook Korean stew, and chat about being at the…

In this episode, @OpenAI Chief Research Officer @markchen90 joins @allenpark to flambé shrimp, cook Korean stew, and chat about being at the frontier of AI research: why scaling laws and pre-training

model-releasesswyx--x

LLMs are getting better at writing GPU kernels. Multi-GPU kernels are the harder test. At @aiDotEngineer World's Fair, @simran_s_arora will …

model-releasestogether-ai--x

LLMs are getting better at writing GPU kernels. Multi-GPU kernels are the harder test. At @aiDotEngineer World's Fair, @simran_s_arora will share ParallelKernelBench, an open-source benchmark built fr

Tools

More on how we're constraining eval environments so that scores better reflect model intelligence: http://cursor.com/blog/reward-hacking-cod…

Cursor discusses methods for improving AI evaluation environments to prevent reward hacking and ensure benchmark scores more accurately measure model capabilities rather than exploitation of evaluatio

toolscursor--x

Riazi-8B: An Urdu Large Language Model for Mathematical Reasoning

arXiv:2606.25568v1 Announce Type: new Abstract: Recent LLMs demonstrate strong mathematical reasoning capabilities, but existing gains rely heavily on English-centric training resources and benchmarks

researcharxiv-cs-cl

Same Evidence, Different Answer: Auditing Order Sensitivity in Multimodal Large Language Models

model-releasesarxiv-cs-cl

arXiv:2606.26079v1 Announce Type: new Abstract: Standard benchmarks for multimodal large language models (MLLMs) score each item on one canonical ordering and miss whether order-irrelevant shuffling c

SurgAtlas: A Large-Scale Surgical Video-Language Dataset with 2,391 Hours of Open and Minimally Invasive Surgery

arXiv:2606.25905v1 Announce Type: new Abstract: We introduce SurgAtlas, the largest surgical video-language dataset to date, comprising 15,291 videos (2,391 hours) spanning 18 surgical specialties and

Tutorials

you may have heard that glm-5.2 at 280 token/s is cool, how about 318 and we still have room to go

tutorialsjeremy-howard--x

GLM-5.2 achieved token generation speeds of 318 tokens per second, surpassing the previously reported 280 tokens per second benchmark, with indications that further performance improvements are possib

5.2 could be better with more RL ...

5.2 could be better with more RL ... Deepswe's benchmark results are my own experience. I've used all models, GLM 5.2 ≈ Claude Opus 4.6–4.7. Kimi 2.7 code more like inference optimization. Looking for

model-releasesollama--x

A Paninian Foundation for Indic Language Processing

arXiv:2606.24172v1 Announce Type: cross Abstract: More than a billion people communicate in Indic languages, yet the natural language processing infrastructure serving them remains fragmented and unde

Local Ai

Accuracy and Satisfaction in Multi-Turn LLM Dialogues for NFR Assessment

arXiv:2606.24834v1 Announce Type: new Abstract: LLM-based dialogue assistants have become mainstream tools for software developers, yet current evaluation benchmarks focus exclusively on functional co

local-aiarxiv-cs-ai

HANCLIP: A Family of Hyperbolic Angular Negation Vision Language Models

arXiv:2606.23843v1 Announce Type: new Abstract: Vision-Language Models (VLMs) are typically pre-trained on large-scale image-text datasets to capture semantic correspondences between visual content an

MERGE: Minimal Expression-Replacement GEneralization Test for Natural Language Inference

arXiv:2510.24295v2 Announce Type: replace Abstract: As many benchmarks have become saturated, it has become increasingly important to create new datasets that evaluate the generalization capacity of c

researcharxiv-cs-cl

ParallelBench: Understanding the Trade-offs of Parallel Decoding in Diffusion LLMs

arXiv:2510.04767v2 Announce Type: replace Abstract: While most autoregressive LLMs are constrained to one-by-one decoding, diffusion LLMs (dLLMs) have attracted growing interest for their potential to

Safety

PixJail: Self-Evolving Paper-to-Pipeline Reproduction for Text-to-Image Jailbreak Evaluation

arXiv:2606.24081v1 Announce Type: cross Abstract: As Text-to-Image (T2I) jailbreak techniques evolve rapidly, existing benchmarks and reproduction workflows often struggle to keep pace. More important

safetyarxiv-cs-ai

Qwen-AgentWorld: Language World Models for General Agents

model-releasesarxiv-cs-cl

arXiv:2606.24597v1 Announce Type: new Abstract: A world model predicts environment dynamics based on current observations and actions, serving as a core cognitive mechanism for reasoning and planning.

RE4: Transformation-aware Imitation of Object Interactions Using Manipulation Modes

arXiv:2606.24403v1 Announce Type: cross Abstract: Object interaction tasks have been a focus of advances in imitation learning. End-to-end methods, dominated by diffusion and flow-based variants have

T2D-Bench: Evidence-Gated Evaluation of LLM Outputs for Type 2 Diabetes Using a Multi-Layer Clinical-Lifestyle Knowledge Graph

arXiv:2606.24145v1 Announce Type: new Abstract: Large language models (LLMs) can produce clinically fluent recommendations for type 2 diabetes while failing to satisfy guideline constraints or explici

The Warrant Gap: Claim-Conditioned Re-scoring for Fact-Checking

arXiv:2606.24627v1 Announce Type: new Abstract: Fact-checking systems built on LLMs achieve high verdict accuracy on standard benchmarks, yet routinely output Supports labels whose cited evidence does

researcharxiv-cs-cl

Try Kimi K2.7 and GLM 5.2 for free in Devin Desktop and CLI

model-releasescognition-ai--x

Try Kimi K2.7 and GLM 5.2 for free in Devin Desktop and CLI Kimi K2.7 Code and GLM 5.2 are available in Devin Desktop and CLI Both perform strongly on FrontierCode Extended, our benchmark for real-wor

When Helpfulness Overrides Causal Caution: Context-Dependent Suppression and Recovery in LLMs

arXiv:2606.24370v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly integrated into decision-support roles in business and policy contexts. While prior benchmark studies have

A Hybrid, Multi-Layered Pipeline for Phishing and Threat Classification: Independently Validated URL and NLP Engines with a Calibrated Multi-Channel Fusion Stage

arXiv:2606.21690v1 Announce Type: cross Abstract: Phishing is a multi-modal threat. We present a hybrid pipeline that scores each modality with its own engine and fuses the results. Three engines are

Chehre: An Emoji-Prompted Video Dataset for Perceptually Diverse Facial Expression Recognition

arXiv:2606.21657v1 Announce Type: new Abstract: Facial expressions are nonverbal social signals used in human interaction, but facial expression recognition datasets often focus on static images, basi

GeoRouteNet: Geometry-Enhanced Non-Autoregressive Neural Solver for the Traveling Salesman Problem

arXiv:2606.22776v1 Announce Type: new Abstract: The traveling salesman problem (TSP) is a canonical NP-hard combinatorial optimization benchmark that tests the representational capacity and generaliza

Hard or Just Unreached? Diagnosing the Sampling Blind Spot in Math-Reasoning Difficulty Estimation

arXiv:2606.19636v2 Announce Type: replace Abstract: Math and science reasoning benchmarks rely on pass@k, the fraction of sampled chains that reach gold, as the canonical per-example difficulty signal

researcharxiv-cs-lg

Hardware

LLMs write fast single-GPU kernels. Ask for a multi-GPU one and they fall apart. ParallelKernelBench (PKB) measures how they fail by benchma…

LLMs write fast single-GPU kernels. Ask for a multi-GPU one and they fall apart. ParallelKernelBench (PKB) measures how they fail by benchmarking against 87 problems pulled from real codebases includi

hardwaretogether-ai--x

MapReason-OSM: Can Vision-Language Models Make Graph-Verifiable Mobility Decisions from Street Maps ?

arXiv:2606.22597v1 Announce Type: new Abstract: Vision-language models (VLMs) are increasingly used to read maps for logistics, delivery, and accessible navigation, where the output is an actionable d

Safety

SAGE: An Expert-Annotated South Asian GI Endoscopy Dataset for Multimodal Learning and Hallucination Analysis

arXiv:2606.22144v1 Announce Type: new Abstract: Gastrointestinal cancers represent a growing health burden in the South Asian region, driven largely by rapid changes in socio-economic conditions & lif

safetyarxiv-cs-cv

Sub-Billion, Super-Frontier: Small Language Models Rival Zero-Shot Frontier LLMs on General and Literary Relation Extraction

arXiv:2606.22606v1 Announce Type: cross Abstract: Large language models (LLMs) achieve strong relation extraction (RE), but their computational demands and reliance on proprietary APIs limit deploymen