Search: “evaluation”

Benchmarking Large Language Models for Biomedical Relation Extraction

arXiv:2609.19071v1 Announce Type: new Abstract: Extracting SNP-phenotype associations from biomedical literature is vital but challenging. We benchmarked diverse NLP models, including MLMs, hybrid arc

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

BOOM: Benchmarking Out-Of-distribution Molecular Property Predictions of Machine Learning Models

DGX agent

arXiv:2505.01912v3 Announce Type: replace-cross Abstract: Data-driven molecular discovery leverages artificial intelligence/machine learning (AI/ML) and generative modeling to filter and design novel

model-releasesarxiv-cs-ai

17 Sept 2026

Clueing up LLMs with Tool-Augmented Deductive Reasoning

arXiv:2609.18736v1 Announce Type: new Abstract: Despite recent advances in large language models (LLMs), performing logically consistent deductive reasoning over extended interactions remains challeng

Collective Loss of Control in LLM Agent Systems: An Epidemic Account of Mutation, Contagion, and Recovery

arXiv:2609.18460v1 Announce Type: new Abstract: How does a multi-agent system evolve from a local deviation into collective loss of control? We propose an epidemic explanation organized around acciden

Tutorials

Come hang out with AI engineers, tech leads, and the Fireworks team during AI Engineer 2026. Want to leave with hands-on experience fine-tun…

Come hang out with AI engineers, tech leads, and the Fireworks team during AI Engineer 2026. Want to leave with hands-on experience fine-tuning, evaluating, and deploying a model on Fireworks? Bring y

tutorialsfireworks-ai--x

Compiled Agency: Frontier General-Purpose Coding Agents Build Winning Game Players from Bare Interaction - from Flappy Bird to StarCraft II and Civilization

arXiv:2609.18996v1 Announce Type: new Abstract: LLM agents have repeatedly struggled to convert knowledge of a game into competent play, even when researchers build the agent around the model - supply

safetyarxiv-cs-ai

Correlation-Guided Encoder Selection for Multi-Encoder Large Audio-Language Models

arXiv:2609.18041v1 Announce Type: cross Abstract: Multi-encoder fusion extends Large Audio-Language Models (LALMs) beyond speech-centric recognition, but selecting encoders via intuition or exhaustive

Decodable but Misrouted: Sparse Features Uncover a Readout Gap in Vision-Language Models for Harmful Meme Detection

arXiv:2609.18860v1 Announce Type: cross Abstract: When a large vision-language model misclassifies a harmful meme, the failure may reflect missing internal evidence or an inability to route represente

Echo: Learning-based Matching Decompilation using Trusted Back Translation

arXiv:2609.18706v1 Announce Type: cross Abstract: Neural decompilers can recover readable and recompilable source code from binaries, but their predictions remain difficult to trust. Matching decompil

Applications

Examining the Difference in Human Behavior Between Virtual and Real-World Human-Robot Teaming

arXiv:2609.18900v1 Announce Type: new Abstract: Prototyping and evaluating human-robot teaming (HRT) scenarios in the real-world is costly. Virtual simulation of HRT scenarios has been adopted as an a

applicationsarxiv-cs-ro

Applications

Extracting Probabilistic Knowledge from Large Language Models for Bayesian Network Parameterization

arXiv:2505.15918v3 Announce Type: replace-cross Abstract: In this work, we evaluate the potential of Large Language Models (LLMs) in building Bayesian Networks (BNs) by approximating domain expert pri

applicationsarxiv-cs-ai

From Pixels to Pairs: A Comprehensive Benchmark of LLM-Based Key-Value Extraction in Noisy Document Settings

arXiv:2609.17538v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used for structured information extraction from documents, yet their behavior under realistic OCR noise re

GraphEcho: Structural Redundancy and Evidence Provenance in LLM Graph Agents

arXiv:2609.17695v1 Announce Type: new Abstract: A large language model (LLM) agent can follow more graph paths without acquiring more independent evidence. GraphEcho tests whether agents mistake these

How a solo founder runs a five-continent tender platform on AlloyDB and MCP

model-releasesgoogle-cloud-ai

Editor's note: Lucius AI, a tender-intelligence startup covering markets across five continents, runs its entire data platform on AlloyDB for PostgreSQL with a single operator. By migrating semantic s

How Much is a Human Right Worth? ECtHR-NPD: A Benchmark for Predicting Non-Pecuniary Damage Awards

arXiv:2609.18908v1 Announce Type: new Abstract: Existing legal benchmarks cover diverse tasks, while continuous monetary remedies remain comparatively underexplored. We introduce ECtHR-NPD, to the bes

iMINDBench: iEEG Multi-Institution Neural Decoding Benchmark

model-releasesarxiv-cs-lg

arXiv:2609.18104v1 Announce Type: new Abstract: Intracranial electroencephalography (iEEG) is widely used to record electrical activity directly from electrodes inside the human brain, making it an at

Introducing Benchmark Reviews: our new initiative to audit AI benchmarks. We are launching with 15 benchmarks: 4 Verified, 9 Flawed, and 2 w…

model-releasesyohei-nakajima--x

Epoch AI has launched “Benchmark Reviews,” an initiative to evaluate existing AI benchmark datasets. The program initially audits 15 benchmarks, categorizing them as 4 verified, 9 flawed, and 2 lackin

Linguistic Triggers of Gender and Racial Bias in Open-Weight LLMs Applied to Recruitment

arXiv:2609.18106v1 Announce Type: cross Abstract: Open-weight large language models are rapidly entering hiring pipelines, yet their discriminatory failure modes -- and the regulatory exposure these c

LM Fight Arena: Benchmarking Large Multimodal Models via Game Competition

arXiv:2510.08928v2 Announce Type: replace Abstract: Existing benchmarks for large multimodal models (LMMs) often fail to capture their performance in real-time, adversarial environments. We introduce

Local Ai

LocQE: Principled Domain Adaptation for Localisation Quality Estimation by Leveraging Post-Edits

arXiv:2609.18720v1 Announce Type: new Abstract: Learned quality estimation (QE) models such as COMETKiwi are widespread and work well for general machine translation evaluation. However, they are know

local-aiarxiv-cs-cl

Playing log(N)-Questions over Wikipedia Abstracts: Communication Efficiency Between Paired Frontier Models

arXiv:2609.19113v1 Announce Type: new Abstract: We evaluate six frontier language models on the two-agent log(N)-Questions game. A questioner sees N Wikipedia lead paragraphs and must identify a secre

QEMScore: How Much Does the Measurement Add to Learned Quantum Error Mitigation?

model-releasesarxiv-cs-lg

arXiv:2609.17896v1 Announce Type: cross Abstract: How much does the noisy measurement add to learned quantum error mitigation? An accuracy table cannot say, because a model handed circuit structure ca

Industry

Reduce time-to-hire for quality candidates with AI-powered Amazon Connect Talent

Amazon Connect Talent is an AI hiring solution built for talent acquisition leaders managing scaled hiring. It delivers AI-led interviews, data-driven assessments, and consistent evaluation, helping r

industryaws-ml-blog

ReFigBench: Benchmarking Scientific Figure Reconstruction as Editable PowerPoint Artifacts

arXiv:2609.18844v1 Announce Type: new Abstract: Multimodal coding agents are expected to turn visual inputs into usable artifacts, and they act through a harness, the layer of tools, context managemen

Reflections on Trusting Trust, Revisited: Contaminating Self-Modifying AI Coding Agents with Poisoned Benchmarks

arXiv:2609.17817v1 Announce Type: cross Abstract: Thompson's 'Reflections on Trusting Trust' showed that a compiler can be poisoned to reinsert its own backdoor, so that even recompiling clean source

SNOMED CT Concept Recommendation from Masked Clinical Context

arXiv:2609.17855v1 Announce Type: new Abstract: Standardizing clinical language to SNOMED CT supports interoperability, analytics, and reusable phenotyping, but concept recommendation remains difficul

Social Laws for Multi-agent Coordination in Stochastic Environments

arXiv:2609.18929v1 Announce Type: cross Abstract: In multi-agent environments, coordinating agents to prevent interference and ensure robust individual performance is a critical challenge. Previous re

safetyarxiv-cs-ai

Technical Feedback: GLM-5.1 Retirement — Significant Regression in Agentic Workloads

Hi Ollama Team, I’m writing regarding the announced retirement of GLM-5.1 from Ollama Cloud on 25 September 2026. I wanted to provide some fairly detailed technical feedback because GLM-5.1 is not sim

model-releasesr-ollama

Which LLM is Best for Translating Natural Language Goals to PDDL

arXiv:2609.18731v1 Announce Type: new Abstract: Bridging the gap between human intent and machine execution remains a challenge in automated planning, where expressing goals in formal languages like P

Zero-Shot Cross-Lingual Recognition of Sign Language Handshapes

arXiv:2609.18772v1 Announce Type: new Abstract: Sign language processing advances rapidly for high-resource languages such as American Sign Language (ASL), yet most of the world's sign languages lack

A panoramic aerodynamic performance prediction method for turbomachinery cascades using transformer-enhanced neural operator

arXiv:2609.16066v1 Announce Type: new Abstract: To enable flexible and rapid aerodynamic performance evaluation in turbomachinery design, this paper proposes a panoramic performance prediction framewo

researcharxiv-cs-lg

AI for Games in the Foundation Model Era

arXiv:2609.16679v1 Announce Type: new Abstract: Foundation models, alongside advances in learned game-world models, are reshaping AI across the game lifecycle. Beyond playing games, recent systems mod

researcharxiv-cs-ai

Autoformalizing Argumentative Material Inferences

arXiv:2609.16991v1 Announce Type: new Abstract: Natural language arguments are compelling before they are formally explicit. A premise supports a claim through defeasible warrants, background commitme

researcharxiv-cs-cl

Benchmarking Factual Robustness of LLMs via Multi-conversation Persuasion

arXiv:2609.16777v1 Announce Type: new Abstract: As Large Language Models (LLMs) increasingly serve as primary knowledge retrieval interfaces, their robustness against extit{persuasion attacks}---attem

safetyarxiv-cs-cl

Bio-Inspired Palette Evolution in Indirectly Encoded Substrates: Timescale Compatibility Shapes Activation Function Discovery

arXiv:2609.17067v1 Announce Type: cross Abstract: Indirectly encoded neural networks can assign different activation functions to individual nodes, but the right functions are rarely known in advance.

researcharxiv-cs-lg

Bridging the Perceptual Gap: Residual-Enhanced Downscaling and Manifold-Aware Perception Alignment Adaptation for NR-IQA

model-releasesarxiv-cs-cv

arXiv:2609.16664v1 Announce Type: new Abstract: Leveraging Large Vision-Language Models like CLIP has recently set new benchmarks for No-Reference Image Quality Assessment (NR-IQA). However, the contr

Competence-Preserving Resume Perturbations Expose Presentation Sensitivity in LLM Screening

arXiv:2609.16517v1 Announce Type: cross Abstract: Resume screeners must infer job-relevant competence from resumes whose presentation can vary substantially in wording, structure, stylistic polish, an

Local Ai

CorrRisk-WM: Corridor-Conditioned Risk World Modeling for Safety-Critical Trajectory Planning

arXiv:2609.16724v1 Announce Type: new Abstract: Safe local planning requires forecasting surrounding-agent motion and evaluating candidate-specific risks, since identical agent motion can pose differe

local-aiarxiv-cs-ro

Applications

Cross-Anatomy Transfer Versus Sparse Interpolation in Digital-Twin-Oriented Aortic Fluid-Structure Interaction Surrogates

arXiv:2609.16322v1 Announce Type: new Abstract: Surrogate credibility for fluid-structure interac- tion (FSI) requires distinguishing transfer across independent anatomies from interpolation within an

applicationsarxiv-cs-ai

Cross-Domain Inference for Human Localization: Applying Wi-Fi RSSI Data to CSI-Trained Models

arXiv:2609.17204v1 Announce Type: cross Abstract: Wi-Fi signal data can be used to compromise the privacy of individuals. While many existing approaches rely on Channel State Information (CSI), collec

researcharxiv-cs-lg

Few-Shot Degradation Is Not What It Seems: Behavioral Evidence, Representation Analysis, and a Random-Text Control Across 12 Models, 2 Tasks, and 2 Architectures

arXiv:2609.15990v1 Announce Type: new Abstract: Few-shot prompting sometimes degrades language models instead of helping them, but why this happens is unknown. We evaluate 12 open-weight models on two

FluxVLA Engine: A One-Stop VLA Engineering Platform for Embodied Intelligence

arXiv:2609.17210v1 Announce Type: cross Abstract: Vision-language-action (VLA) models, world-action models (WAMs), and offline reinforcement learning methods are rapidly expanding the design space of

safetyarxiv-cs-ai

GeoCrossBench: Cross-Band Generalization for Remote Sensing

model-releasesarxiv-cs-lg

arXiv:2511.02831v2 Announce Type: replace Abstract: The data for remote sensing is constantly acquired, and new data comes from a growing number and diversity of satellites, while the vast majority of

Google DeepMind cofounder Shane Legg warns that advancing AI must never run ahead of safety and launches the DeepMind Institute to explore the deployment of AGI (Financial Times)

Financial Times: Google DeepMind cofounder Shane Legg warns that advancing AI must never run ahead of safety and launches the DeepMind Institute to explore the deployment of AGI — Shane Legg launches

model-releasestechmeme

Google will now let any AI agent run your smart home

model-releasesthe-verge-ai

Google is opening up its smart home to AI agents, letting tools like Claude and Open Claw access and control your connected devices and analyze your home's data using the standardized Model Context Pr

GRADE: Benchmarking Discipline-Informed Reasoning in Image Editing

model-releasesarxiv-cs-cv

arXiv:2603.12264v2 Announce Type: replace Abstract: Unified multimodal models target joint understanding, reasoning, and generation, but current image editing benchmarks are largely confined to natura

Agents

Improving HCLS AI reasoning with open-source agent skills

AI agents on foundation models often misapply healthcare and life sciences decision frameworks, citing the right guideline but applying it incorrectly. This post shares 38 open-source agent skills acr

agentsaws-ml-blog

Latent Undertow: How Ordinary Typos Break Probes

arXiv:2609.15994v1 Announce Type: new Abstract: LLMs handle ordinary typing variation fluently: a typo or missing punctuation leaves both user intent and the model's response substantively unchanged.