Search: “benchmark”

BanglaTurn: A Benchmark and Whisper-Based Model for End-of-Turn Detection in Bangla Speech

arXiv:2609.29371v1 Announce Type: new Abstract: This paper presents BanglaTurn, a corpus for end-of-turn detection in Bangla conversational speech, and a model trained on it. The corpus holds 35,374 s

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

fable.intermittent: benchmarking probabilistic forecasting methods for intermittent time series

DGX agent

arXiv:2609.28607v1 Announce Type: new Abstract: Intermittent time series are common in spare-parts demand and retail sales. Since the cost of forecast errors is typically asymmetric, decisions such as

model-releasesarxiv-cs-lg

25 Sept 2026

Local Ai

I benchmarked tool-call reliability on qwen2.5-coder against pi, Hermes, and Goose - all three got 0%

A while back I posted here about a CLI agent I'm building that parses tool calls out of a model's plain-text output instead of trusting native function-calling to work. Finally ran the actual numbers.

local-air-ollama

PHOSA: Photorealistic 3D Sign Avatar Modeling and Benchmark

arXiv:2609.29292v1 Announce Type: new Abstract: In this work, we focus on photorealistic sign avatar modeling, which is crucial for effective communication with the Deaf community and is characterized

When Identical Rows Disagree: From Benchmark Identifiability to Replication-Robust Anomaly Detection

model-releasesarxiv-cs-lg

arXiv:2609.29580v1 Announce Type: new Abstract: A released table is often treated as an i.i.d. sample, although its repeated rows may encode business frequency, repeated entities, joins, resampling, o

ThinkingCap 3.8-27B vs. Swift 3.8-27B vs. Qwen 3.8-27B Benchmarks

With the release of ThinkingCap-Qwen3.8-27B, I thought it would be worthwhile to do a comparison between the original Qwen3.8-27B, the new ThinkingCap, and Swift-Qwen3.8-27B. Both Swift which I alread

24 Sept 2026

Compared to What? A Human-Anchored Security Benchmark for LLM-Generated Infrastructure-as-Code

model-releasesarxiv-cs-ai

arXiv:2608.28021v2 Announce Type: replace-cross Abstract: Large language models increasingly author Infrastructure-as-Code (IaC), where one insecure default is provisioned straight into production. Pr

23 Sept 2026

Deep Generative Crystal Structure Prediction: A Benchmark Study and a Controlled Test of Prototype Dependence

model-releasesarxiv-cs-lg

arXiv:2609.26502v1 Announce Type: cross Abstract: Deep generative models are widely reported to enable de novo crystal structure prediction (CSP), but their capability has not been measured consistent

23 Sept 2026

Radiomics--Foundation Fusion for Interpretable RCC Classification: Internal Benchmarking and Exploratory External Transfer

arXiv:2609.26578v1 Announce Type: new Abstract: Accurate preoperative subtype classification of renal cell carcinoma (RCC) from contrast-enhanced CT remains clinically challenging because clear cell R

researcharxiv-cs-cv

23 Sept 2026

Tutorials

Benchmarking Hybrid Deep Learning Architectures for Predictive Maintenance in Industry 4.0

arXiv:2609.22583v1 Announce Type: new Abstract: Predictive maintenance in Industry 4.0 refers to using data from sensors, machines, and production systems to estimate when equipment is likely to fail,

tutorialsarxiv-cs-lg

Benchmarking Off-the-Shelf Multimodal AI Models Against Dermatologists on Patient-Captured Skin Images

arXiv:2609.24190v1 Announce Type: new Abstract: Artificial intelligence (AI) has advanced at a rapid pace in recent years. Initially, breakthroughs in large language models caught widespread attention

MuLA-Bench: A Multilingual Long-Form Audio Understanding Benchmark via Multi-Tier Auditing

arXiv:2609.23416v1 Announce Type: cross Abstract: Long-form audio performance is often summarized by context length and aggregate accuracy, obscuring how language, evidence, and task jointly shape dif

Opus 5.5 made me curse at Claude less, which is an unfortunate benchmark to now be using but a win.

model-releasesallie-k--miller--x

On 22 September 2026, Allie K. Miller posted on Twitter that the Opus 5.5 model led her to curse at Claude less frequently. She notes that while reducing profanity toward another AI is an undesirable

Vision Transformers versus convolutional neural networks for fine-grained orchid genus identification in a species-rich, data-poor flora: a controlled benchmark on the Orchidaceae of New Guinea

arXiv:2609.24064v1 Announce Type: new Abstract: New Guinea is the world's richest island flora (~2,856 orchid species), yet most species are represented by only a handful of photographs, far fewer tha

Benchmarking Gender Bias in Machine Translation Evaluation Metrics across Occupations

arXiv:2609.21490v1 Announce Type: new Abstract: Gender bias remains a persistent concern in machine translation (MT), affecting both generated translations and their automatic evaluation. When a sourc

safetyarxiv-cs-cl

21 Sept 2026

Beyond Kinematics: Benchmarking Simulation Fidelity for Muscle-Driven Imitation Learning

arXiv:2609.21909v1 Announce Type: new Abstract: In this work, we conduct a systematic comparison of two state-of-the-art motion-imitation reinforcement learning (MIRL) pipelines, one built on SCONE/Hy

safetyarxiv-cs-lg

21 Sept 2026

Applications

Before the Arrest: Benchmarking LLMs on Criminal Profiling from Incomplete Evidence

arXiv:2609.19965v1 Announce Type: new Abstract: Large Language Models (LLMs) are increasingly applied to legal and criminal justice tasks, yet existing work focuses almost exclusively on post-arrest s

applicationsarxiv-cs-cl

18 Sept 2026

Is It Still Worth Training a Classical Model in the Era of LLMs? A Crossover Benchmark on Tabular Data

model-releasesarxiv-cs-ai

arXiv:2609.20218v1 Announce Type: cross Abstract: Large language models can label a tabular row from a plain-English description with no training - a capability now shipping in mainstream spreadsheet

18 Sept 2026

Benchmarking Tabular Foundation Models as Surrogates in Expensive Evolutionary Optimization

arXiv:2609.18130v1 Announce Type: cross Abstract: Surrogate-assisted evolutionary algorithms (SAEAs) are effective methods for solving expensive optimization problems (EOPs), where surrogate models re

researcharxiv-cs-lg

17 Sept 2026

Prosthesis-Aware 3D Human Pose Estimation: A Dataset and Benchmark for RSP Users

arXiv:2609.18406v1 Announce Type: new Abstract: Recovering 3D human body motion from video is important for applications such as rehabilitation assessment and sports performance evaluation. For prosth

17 Sept 2026

A Sentinel-2 benchmark dataset for deep-learning active-fire segmentation across 25 California wildfires

arXiv:2609.16199v1 Announce Type: cross Abstract: This article describes an open image dataset for developing and evaluating active-fire segmentation methods in satellite imagery. The dataset contains

16 Sept 2026

Benchmarking Factual Robustness of LLMs via Multi-conversation Persuasion

arXiv:2609.16777v1 Announce Type: new Abstract: As Large Language Models (LLMs) increasingly serve as primary knowledge retrieval interfaces, their robustness against extit{persuasion attacks}---attem

safetyarxiv-cs-cl

16 Sept 2026

Lesion-centered 3D mapping of colonoscopy procedures: validation of a hierarchical ensemble pipeline on public benchmark videos

arXiv:2609.16672v1 Announce Type: new Abstract: Background and Objective: Colonoscopy recording practice preserves text reports and still photographs, while the spatial information already present in

16 Sept 2026

Duplicating baseline benchmarks [D]

researchr-machinelearning

Suppose I create two machine learning models suppose tree and neural network for a task let's suppose regression problem, now suppose I am sending both of this paper to two different journals, now the

14 Sept 2026

MAxBench: A Multinomial Concept Recovery Benchmark

arXiv:2609.13072v1 Announce Type: cross Abstract: Fine-grained control of language model behaviors (e.g., steering) is among the more actionable outcomes of interpretability research. For binary conce

14 Sept 2026

How Much Velocity Does Off-Ball Space Value Need? A Broadcast-Viewport Benchmark

arXiv:2609.10801v1 Announce Type: new Abstract: Velocity-aware pitch control is standard, but under a broadcast viewport half the players are off screen and on-screen velocities come from a drifting c

11 Sept 2026

I built a thing that benchmarks Ollama models on your own hardware and ranks them

I kept picking local models badly. Parameter count, quantization, VRAM. I could read all of it and still not know whether a model would be any good on my machine. So I stopped guessing and wrote somet

model-releasesr-ollama

8 Sept 2026

New Benchmark: The Struggle Bench

How it works. The model being tested is given a server capable of running it's weights and full context. That server is placed in a median priced apartment. The AI is given a bank account with for ren

7 Sept 2026

4xRadeon AI Pro R9700 people, how are your benchmarks?

Was considering choosing a 4xRadeon AI Pro R9700 build for a office AI server. Curious to know what results you guys are getting? Mainly considering DeepSeek V4 flash and Qwen 3.8 Flash with offloadin

6 Sept 2026

Beyond BLEU: A Case for Redefining Sign Language Translation Benchmarks

arXiv:2609.03734v1 Announce Type: cross Abstract: BLEU-4 is the standard metric for evaluating sign language translation (SLT), but spoken-language metrics may not adequately reflect sign language pro

researcharxiv-cs-ai

4 Sept 2026

MMPCBench: Benchmarking Multimodal Large Language Models on Proactive Critique of Flawed Inputs

arXiv:2608.29286v1 Announce Type: new Abstract: As Multimodal Large Language Models (MLLMs) evolve into sophisticated interactive assistants, their reliability depends not only on following instructio

safetyarxiv-cs-ai

1 Sept 2026

Hardware

Benchmarking large language model agent societies against human behavioural distributions

arXiv:2608.28182v1 Announce Type: cross Abstract: Populations of large language model agents are increasingly used as experimental societies. Three doubts shadow every such result: whether the agents

hardwarearxiv-cs-cl

31 Aug 2026

Classical and Hybrid Quantum Machine Learning for Trigger-Like Event Selection on CMS Open Data: An Eight-Qubit, PCA-Constrained Benchmark

model-releasesarxiv-cs-lg

arXiv:2608.26224v1 Announce Type: cross Abstract: Event triggering sits at the heart of high-energy physics, where the rare events of interest must be retained while an overwhelming background is disc

28 Aug 2026

Fusing Perceptual Vision Experts with Multimodal Large Language Models for Explainable Plant Disease Diagnosis: From Benchmark Imagery to Real-World Robotic Field Validation

arXiv:2608.24934v1 Announce Type: new Abstract: Accurate field plant disease diagnosis requires reliable fusion of uncertain and conflicting perceptual evidence. We present the Hybrid Hierarchical Mul

27 Aug 2026

Local Ai

Benchmarking local agent architecture (ReAct loop vs. Graph)

Hi all! I've been building a local agent harness, but before writing the code, I wanted to get some intuition around whether (and how) structuring agent execution as a graph helps vs. standard tool-ca

local-air-ollama

26 Aug 2026

Scientific Image Synthesis: Benchmarking, Methodologies, and Downstream Utility

arXiv:2601.17027v2 Announce Type: replace-cross Abstract: While synthetic data has proven effective for improving scientific reasoning in the text domain, multimodal reasoning remains constrained by t

researcharxiv-cs-ai

26 Aug 2026

Agents

TrustShiftProbe: Characterizing, Benchmarking, and Defending Staged Trust Attacks on MCP Servers

arXiv:2608.23763v1 Announce Type: cross Abstract: The Model Context Protocol (MCP) has emerged as the standard layer connecting Large Language Model agents to external tool backends. This openness int

agentsarxiv-cs-ai

26 Aug 2026

Benchmarking Retrieval-Augmented Generation Strategies for Large Language Model-Based Travel Mode Choice Prediction

arXiv:2508.17527v2 Announce Type: replace Abstract: Accurately predicting travel mode choice is essential for effective transportation planning, yet traditional statistical and machine learning models

researcharxiv-cs-ai

Definitional Sensitivity in Media Bias Detection: A Multi-Definition Dataset and Benchmark

arXiv:2608.23095v1 Announce Type: new Abstract: Media bias detection relies on definitions and examples that specify what counts as bias, yet these specifications often vary across datasets or remain

KAN-Robust-Bench: A Benchmark for Evaluating the Robustness of Kolmogorov-Arnold Networks

model-releasesarxiv-cs-ai

arXiv:2608.21488v1 Announce Type: cross Abstract: While machine learning models have demonstrated strong performance in many domains, these models have shown profound vulnerabilities when they are exp

No One Model Catches Every Harm: Benchmarking Content Moderation Across Safety Scenarios

arXiv:2608.21775v1 Announce Type: new Abstract: Large Language Models (LLMs) are increasingly deployed in real-world applications, yet they remain vulnerable to generating harmful content. From advers

safetyarxiv-cs-cl

Noise Floor Audit for Agent Benchmarks

arXiv:2608.22331v1 Announce Type: new Abstract: We audit measurement variability for 3 native tool-calling endpoints across 2 providers on the official BFCL multiple and parallel categories, using mat

[Benchmark] llama.cpp batch/ubatch impacts on PP and TG

My test is running DeepSeek v4 Flash 0731 at native size on DGX Spark machine (GB10, 128 GB unified memory). The model size is bigger than RAM, so weights will be loaded many times when running. To im

24 Aug 2026

Multilingual Verifier Bias in RLVR: Benchmark, Rollout Diagnosis, and the Cross-Lingual Selection Bottleneck

arXiv:2608.20362v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) is a standard recipe for training large language models on mathematical reasoning, where an answer

24 Aug 2026

OmniAssistBench: Assistant-style Interaction Benchmark for Omni-LLMs

arXiv:2608.21360v1 Announce Type: new Abstract: Recent omni-modal large language models (Omni-LLMs) show great potential as real-time video assistants, which continuously perceive environments and gui

24 Aug 2026

Benchmarking qwen3.8:27b by having it port a 34k line C game to one HTML file. The agent mattered more than I expected.

My hobby project skill-issue is an arena FPS in a single C file, about 34k lines, everything procedural. My test for local setups is to have an agent port it to a single file HTML version with three.j

model-releasesr-ollama

22 Aug 2026

Been tweaking my Qwen 3.8 setup, up to 45+ steady T/ps at 8bit quant. Realised I'm now top T/ps for this model+ctx across all benchmarked M-series chips. Full args linked below, happy to discuss as this was a pain of trial and error.

https://omlx.ai/benchmarks/performance/2pko3m1k - you can expand the raw args, but I have full annotations of what worked and what didn't. I'm now testing the model on acutal coding and haven't seen a

20 Aug 2026

PLASMA: A Layout-Aware Benchmark Reveals Memory Layout Matters for Graph-based ANNS on GPU

arXiv:2508.15436v2 Announce Type: replace-cross Abstract: We propose a extbf{P}latform for extbf{L}ayout-extbf{A}ware extbf{S}earch and extbf{M}emory extbf{A}rrangement (extbf{PLASMA}), a unified eval