Search: “benchmark”

SynthGait-19K: A Physically Grounded Synthetic Video Dataset for Gait Parameter Estimation

arXiv:2609.08108v1 Announce Type: new Abstract: Accurate estimation of clinically meaningful gait parameters from monocular video is important for scalable mobility assessment, yet progress is limited

9 Sept 2026

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Vision-Encoder Behavioral Fingerprints of Image-to-Image Generative Models: A Training-Paradigm-Driven Taxonomy of Six Commercial APIs

DGX agent

arXiv:2606.14787v2 Announce Type: replace Abstract: We study six production image-to-image AI systems (gpt-image-1, Gemini 2.5 Flash Image, Flux Kontext, SDXL img2img, SD3 img2img, and Qwen Image Edit

model-releasesarxiv-cs-cv

9 Sept 2026

I reduced image-processing token usage by ~95% compared with GPT-4o direct vision, while maintaining roughly the same accuracy.How significant is that?[P]

model-releasesr-machinelearning

I'm testing a new approach for reducing the cost of image-based LLM inference. I evaluated it on the MOMA Graph benchmark, using 1,315 questions. Compared with using GPT-4o to process the original ima

8 Sept 2026

so it was the harness not the model. more sneaky games with unacknowledged domain-specific neurosymbolic AI doing the real work.

model-releasesgary-marcus--x

so it was the harness not the model. more sneaky games with unacknowledged domain-specific neurosymbolic AI doing the real work. OpenAI published 99.9%. The benchmark's own software returned 62.7%. Sa

8 Sept 2026

AVENUE: Audio-Video EditiNg Understanding and Evaluation

arXiv:2609.04253v1 Announce Type: cross Abstract: Audio-video (AV) editing aims to modify audio and video content according to a target prompt. Unlike single-modality editing, AV editing requires mode

ICM-Bench: Person-Level Identity Reasoning in Multimodal Agents with Long-Term Memory

arXiv:2609.04438v1 Announce Type: new Abstract: Long-horizon multimodal agents should remember not only what happened but also who participated. This capability depends on linking recurring faces, voi

Last Week in AI #343 - GPT-6, OpenAI’s agents chatted on a wiki, Fable 5.1

model-releaseslast-week-in-ai

OpenAI released GPT‑6 “Astra” — a model benchmarked for superior computer navigation, coding and complex math that can book DMV appointments and hunt apartments faster than an average user. The rollou

LLM-guided program evolution improves 10 best-known circle-packing solutions (Packomania csqv, N=101-114) [R]

model-releasesr-machinelearning

I used an LLM to iteratively evolve an optimization algorithm rather than solve the packing directly. Starting from a simple seed solver, the LLM proposes algorithmic changes guided by a scoreboard of

Phase Transition Frequency as a Training Time Predictor of Test Accuracy in ResNets

arXiv:2609.05194v1 Announce Type: cross Abstract: The number of discrete class-separability jumps observed during ResNet finetuning is examined empirically as a predictor of final test accuracy. Acros

Physics-Aware Random Walk Fingerprints for Scalable Power Grid Graph Classification

model-releasesarxiv-cs-lg

arXiv:2609.04943v1 Announce Type: new Abstract: Recent benchmarks such as PowerGraph provide large collections of power-grid graphs for cascading-failure classification. Graph neural networks (GNNs) a

DeepSeek-V4-Flash-Vision Q8 vs Qwen3.8-Flash-Next Q8

I'm using DS-V4-Flash-Vision with Q8_K_XL quantization locally as my everyday engine, and for some time now I've been doing a lot of comparisons with Qwen3.8-Flash-Next, also with Q8_K_XL quantization

6 Sept 2026

Planning to get a cheap-ish GPU. Would appreciate some advice.

Hi, I've been wanting to run my own local LLM for some time and I finally saved enough to get a budget GPU. I can spend 700 at most, and I'm looking for a GPU that can run quantized 30B~ parameter mod

6 Sept 2026

Beyond Majority Vote: Multi-Perspective Adjudication for Medical Hallucination Detection

arXiv:2609.03953v1 Announce Type: new Abstract: Understanding the frequency of factual errors in chatbot-generated text and evaluating systems that detect these errors is critical for determining chat

CORE: Improving Compositional Reasoning in MLLM Embedding via Reranker Distillation

arXiv:2609.04083v1 Announce Type: cross Abstract: MLLM-based embedding models remain limited in compositional retrieval, often failing to distinguish scenes containing the same concepts but different

FailBench: How Reliable are VLMs at Judging Robot Task Success?

arXiv:2609.03611v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) are increasingly used to evaluate robot manipulation outcomes, but existing benchmarks offer limited evidence of cross-d

Has anyone already tried IFM's new K2-Horizon-MoVA-36B-A4B?

How good/bad is it against comparable MoEs the same size? How does it compare against Qwen 3.6 35BA3B? Since we don't have 3.8 35B this seems like an upgrade if we look at some benchmarks like termina

How is AI really affecting your job? @Cohere_Labs released the Agentic Task Ecosystem, a new dataset of 690K+ tools built for AI agents. Und…

How is AI really affecting your job? @Cohere_Labs released the Agentic Task Ecosystem, a new dataset of 690K+ tools built for AI agents. Under a test of whether a tool could independently complete an

model-releasescohere--x

IDSPACE: A Novel Document Generator for Reliable Evaluation of Digital Identity Verification Systems [Extended Technical Report]

arXiv:2609.03052v1 Announce Type: new Abstract: As services move online, trust institutions such as banks, lenders, and governments must verify the identity of remote users. Fraud detection tools are

Mind the Gap: Robustness Risks in PII Detection Systems

model-releasesarxiv-cs-lg

arXiv:2609.03464v1 Announce Type: new Abstract: Personally Identifiable Information (PII) detection is a foundational component of data protection infrastructure where missed entities constitute direc

MINERVA: How Small Can a Manipulation Policy Be and Still Solve LIBERO?

model-releasesarxiv-cs-ro

arXiv:2609.03715v1 Announce Type: new Abstract: Vision-language-action (VLA) models with billions of parameters now dominate the LIBERO manipulation benchmark, but the model capacity actually required

PatchBench: Evaluating AI Agents for Vulnerability Patching

arXiv:2609.04075v1 Announce Type: cross Abstract: AI agents have recently demonstrated strong performance in automated vulnerability patching. However, existing evaluations often validate a patch only

Rent-a-RAG: Embedding-Space Watermarks for Auditing Third-Party RAG

arXiv:2609.03749v1 Announce Type: cross Abstract: Third-party retrieval-augmented generation (RAG) marketplaces create a new auditing problem: data providers may license corpora to a RAG operator, yet

Restricted Eigenvalues Beyond Gaussian Width: Threshold Occupancy under Heavy Tails

model-releasesarxiv-cs-lg

arXiv:2609.03504v1 Announce Type: new Abstract: Restricted eigenvalue (RE) bounds govern stable recovery by norm-regularized estimators. For isotropic sub-Gaussian measurements, the benchmark sample s

SVG-Score: Human-Aligned Evaluation of Text-to-SVG Generation

arXiv:2609.03806v1 Announce Type: new Abstract: Scalable Vector Graphics (SVG) generation is attracting increasing attention as generative models improve in expressiveness and controllability. Progres

Agents That Model Agents: Five Principles Toward a Theory of Mind for 6G Networks

arXiv:2609.01779v1 Announce Type: cross Abstract: Future 6G networks will rely on Large Language Model (LLM) agents to manage the Radio Access Network (RAN). However, current architectures assume inte

C^{3}T: Counterfactual Causal Reasoning for Sentiment Shifts in Social-Media Conversation Trees

arXiv:2609.02131v1 Announce Type: cross Abstract: Sentiment in social-media threads does not only vary across posts; it shifts as users react to claims, corrections, evidence, and hostility within a b

CoViT: Instance-Correspondence Contrastive Learning for Vision Transformer

arXiv:2609.01787v1 Announce Type: new Abstract: Vision Transformers (ViT) excel in semantic understanding but fail to discriminate between object instances (e.g., identical embeddings for two dogs), l

DeepSeek-V4-Flash vs. GLM-5.3-Flash on 2× DGX Spark

I've tried both and been having this debate with myself for the last few days, on two Asus Ascent GX10s (effectively the same as 2x DGX Spark): DeepSeek-V4-Flash-0731 (official weights) GLM-5.3-Flash

Evaluating the Evaluator: Summarization Metrics and LLM-Judges beyond English

arXiv:2503.17039v3 Announce Type: replace-cross Abstract: Automatic text summarization relies on automatic evaluation to quickly determine the quality of summarization models via automatic metrics and

ExecRetrieval: Measuring the Functional-Correctness Gap in Code-Embedding Retrieval

arXiv:2609.01865v1 Announce Type: cross Abstract: Embedding-based code retrieval is a core component of coding agents and retrieval-augmented code generation, where retrieving correct code matters mor

Post-Training Language Models for Gold-Medal Performance in Coding Competitions

arXiv:2609.02849v1 Announce Type: cross Abstract: Competitive programming has become a key test of large language model reasoning, with international competitions such as IOI and ICPC representing its

We built an open-source, model-neutral agent harness and compared it with claude managed agents - for the same model, got same accuracy, upto 75% lower cost

We have been working on an open-source, model-neutral agent harness for general purpose agents called TrueForge, and wanted to understand how much the harness itself actually matters. So we ran 14 tas

AutoXRD: Autonomous LLM Agents and Comprehensive Evaluation for Powder Diffraction Analysis

arXiv:2609.00070v1 Announce Type: cross Abstract: Powder X-ray diffraction (XRD) is central to materials characterization, yet reliable end-to-end automation remains challenging. An XRD agent must int

Local Ai

Calibration is the Bottleneck: An Action-Class Diagnostic of Multi-Turn Tool-Calling

arXiv:2609.00949v1 Announce Type: cross Abstract: Multi-turn tool calling is a core evaluation scenario for large language model (LLM) agents. On public tool-calling benchmarks, open-weight models now

local-aiarxiv-cs-ai

Can Scene Text Recognition Read Rare Compositions?

arXiv:2609.00816v1 Announce Type: new Abstract: Scene text recognition is reported as 89--97% accurate on the six standard benchmarks, and the problem is widely treated as saturated. We present an alt

ClinTraceBench: Source-Verifiable Longitudinal Clinical Reasoning over EHR-Derived Dialogues

arXiv:2609.01111v1 Announce Type: new Abstract: Clinical LLM assistants must reason over multi-visit patient trajectories, yet whether the compact history representations used to scale them---retrieva

DAGGER: Distractor-Aware Graph Generation for Executable Reasoning in Math Problems

arXiv:2601.06853v3 Announce Type: replace Abstract: Chain-of-Thought (CoT) prompting is widely adopted for mathematical problem solving, including in low-resource languages, yet its behavior under irr

Evaluating Style-Personalized Text Generation: Challenges and Directions

arXiv:2508.06374v4 Announce Type: replace Abstract: With the surge of large language models (LLMs) and their ability to produce customized output, style-personalized text generation--'write like me'--

First local-LLM tuning attempt: Qwen3.8-27B true Q4_K_M at 13.2 tok/s near 50-61K context on RTX 5080 16GB

This was my first serious attempt at tuning a local LLM. I started because Qwen3.8-27B IQ3 was fast on my RTX 5080 but the coding quality disappointed me, and the Q4 profiles I tried in LM Studio were

Fleets Need a Context Plane: Rethinking Cooperative Perception for Autonomous Drones

model-releasesarxiv-cs-ro

arXiv:2609.00659v1 Announce Type: new Abstract: Cooperative perception allows a drone fleet to combine observations from multiple viewpoints. However, existing systems typically fix their feature-shar

Good Memory Has ECC: Evaluating the Memory of Vision-Language Models Beyond Accuracy

arXiv:2609.00103v1 Announce Type: cross Abstract: Memory is widely viewed as an important unsolved problem for LLMs and VLMs, and current benchmarks typically evaluate it by testing accuracy over long

Incremental Risk Assessment of Progressive Elder Financial Scams via Instruction-Tuned Small Language Models

arXiv:2609.00005v1 Announce Type: new Abstract: Financial scams targeting older adults increasingly occur through text and voice channels such as email, SMS, and phone calls, unfolding over multiple c

Learning to Remember: End-to-End Training of Memory Agents for Long-Context Reasoning

arXiv:2602.18493v2 Announce Type: replace-cross Abstract: Long-context LLMs and Retrieval-Augmented Generation defer state tracking and evidence consolidation to query time, which is brittle when fact

Research

The Visual Insensitivity Gap: Diagnosing When Vision-Language Models Fail to Use Visual Evidence

arXiv:2609.00868v1 Announce Type: cross Abstract: Vision-language models are evaluated by aggregate accuracy on multimodal benchmarks, a practice that implicitly assumes the model uses its visual inpu

researcharxiv-cs-cl

AdaPath: Query-Adaptive Path-Finding via Path-Bank for Multi-Hop Implicit Biomedical KGQA

arXiv:2608.30556v1 Announce Type: new Abstract: Path-finding over knowledge graphs has become an effective way to ground LLM reasoning on multi-hop questions. However, biomedical QA introduces two dis

Arkios: An Open Bilingual English-Nepali Language Model Trained From Scratch, with a Devanagari-Aware Tokenizer

arXiv:2608.30092v1 Announce Type: cross Abstract: We present Arkios, a 1.04B-parameter dense transformer pretrained from scratch on 150B tokens of bilingual English-Nepali text, using a custom single-

Beyond Correctness: Validity-Oriented Evaluation of Biomedical LLM Judges

arXiv:2608.29127v1 Announce Type: new Abstract: We propose a scalable, validity-oriented pipeline for evaluating biomedical LLM judges when high-quality human judgments are scarce. First, we augment e

Beyond Global Realism: Virtual Try-On Evaluation and Optimization with Dimension-wise Garment Fidelity Assessment

arXiv:2608.29804v1 Announce Type: new Abstract: Virtual try-on (VTON) requires not only realistic generation but also faithful preservation of garment characteristics. However, existing evaluation met