Knowledge catalogue
arXiv:2609.39284v1 Announce Type: cross Abstract: While large language models have achieved remarkable success in isolated code generation, authentic software engineering requires sustained reasoning,
arXiv:2609.38897v1 Announce Type: cross Abstract: Far-field automatic speech recognition(ASR) degrades under reverberation, noise, and talker motion, yet the benchmarks that drive model selection emph
arXiv:2609.35596v2 Announce Type: replace-cross Abstract: Self-evolving LLM agents have gained prominence for their ability to improve after deployment by modifying their harness, including their cont
arXiv:2609.39189v1 Announce Type: new Abstract: Trustworthy Legal AI requires systems that can answer legal questions while grounding their responses in authoritative sources. However, existing Vietna
arXiv:2609.40031v1 Announce Type: cross Abstract: Digital image watermarking is increasingly critical in media contexts, as emerging regulations and industry practices require marking AI-generated con
arXiv:2609.36039v1 Announce Type: cross Abstract: Machine learning (ML) and deep learning (DL) have dominated Intrusion Detection System (IDS) research in recent years. Unfortunately, many existing st
arXiv:2609.32248v1 Announce Type: new Abstract: Large language model (LLM) leaderboards compare model capabilities by ranking models according to their mean performance on fixed benchmarks. However, v
arXiv:2609.35113v1 Announce Type: new Abstract: Symbolic regression (SR) seeks concise and interpretable mathematical expressions from data for scientific equation discovery. Existing SR benchmarks fa
arXiv:2609.30489v1 Announce Type: new Abstract: Large Language Models (LLMs) have demonstrated historic breakthroughs in general reasoning with early successes in biomedical science. However, existing
arXiv:2609.31507v1 Announce Type: new Abstract: Urban uncrewed aerial vehicle (UAV) vision-language navigation (VLN) requires agents to follow instructions across extended urban spaces, inherently dem
arXiv:2609.30971v1 Announce Type: new Abstract: Embodied agents offer a promising route to automating scientific experimentation, yet their progress is constrained by the lack of reliable and systemat
arXiv:2609.29934v1 Announce Type: new Abstract: Does progress on spatial reasoning benchmarks translate into better navigation? Existing benchmarks test isolated inferences from images or videos, with
Recommended benchmark. I expect voice to become one of the main ways people interact with robots and physical AI. That makes speech recognition in real conversations much more important than it seems
arXiv:2607.22115v2 Announce Type: replace-cross Abstract: Given a database S and a natural language question Q, text-to-SQL systems aim to generate an SQL query that correctly answers Q when executed
Built this because I got tired of guessing whether a model would fit in my 6GB VRAM before downloading it, and because ollama.com's library doesn't let you sort by size. What it does: - Benchmarks Oll
arXiv:2609.27395v1 Announce Type: new Abstract: Compact vision-language models (VLMs) now power a growing share of multimodal applications. The benchmarks used to compare them, however, inherit a fron
Claude Opus 5.5 is now available as the Standard effort level in Perplexity Computer. On our WANDR benchmark, it scored 0.610 at $4.13 per task, slightly outperforming Claude Fable 5.1 while costing 6
Retrieval benchmarks sometimes feel benchmaxxed by models, so we wanted to find a way to tie it as close as possible to my objective: finding the article that answers a product question right. We work
arXiv:2609.18994v1 Announce Type: new Abstract: We present the first benchmark suite for formal verification of Programmable Logic Controller (PLC) programs that combines controlled ground truth with
arXiv:2609.18203v1 Announce Type: new Abstract: Human values are deep motivational orientations that shape human behaviors. In e-commerce, they reveal the stable drivers behind users' purchase decisio
arXiv:2609.19071v1 Announce Type: new Abstract: Extracting SNP-phenotype associations from biomedical literature is vital but challenging. We benchmarked diverse NLP models, including MLMs, hybrid arc
arXiv:2505.01912v3 Announce Type: replace-cross Abstract: Data-driven molecular discovery leverages artificial intelligence/machine learning (AI/ML) and generative modeling to filter and design novel
arXiv:2609.17688v1 Announce Type: new Abstract: Wearable assistants require episodic memory over egocentric video, yet current vision-language models face bounded frame budgets, growing visual-token c
arXiv:2609.18644v1 Announce Type: new Abstract: Fallacy-detection benchmarks pair fallacy classes with a single 'valid' or 'none' class that takes everything data collection did not label as a fallacy
arXiv:2609.17817v1 Announce Type: cross Abstract: Thompson's 'Reflections on Trusting Trust' showed that a compiler can be poisoned to reinsert its own backdoor, so that even recompiling clean source
arXiv:2609.17843v1 Announce Type: cross Abstract: Video anomaly detection (VAD) is an actively studied task, having wide applications in typical scenarios such as public surveillance and road traffic
arXiv:2609.19072v1 Announce Type: new Abstract: Large language models are increasingly used for content moderation, but most evaluations still report aggregate accuracy on individual benchmarks. We in
arXiv:2609.18493v1 Announce Type: new Abstract: Semantic labels for indoor mobile laser scanning (MLS) frames remain largely absent from current point cloud semantic segmentation benchmarks, which mai
arXiv:2609.18521v1 Announce Type: cross Abstract: Speech retrieval has become increasingly important as spoken content continues to grow across meetings, lectures, podcasts, and videos. Existing bench
arXiv:2609.16305v1 Announce Type: new Abstract: Large language model (LLM) agents increasingly operate over long-horizon interactions involving tool use, persistent state, evolving authorization, and
arXiv:2609.17128v1 Announce Type: new Abstract: Comprehensive structured data on inter-firm relationships is often scarce or inaccessible because many relationships are privately negotiated, selective
arXiv:2609.17112v1 Announce Type: new Abstract: Large video models have exhibited impressive performance on a wide range of visual question answering tasks, owing to the rise of powerful, pretrained t
arXiv:2609.16614v1 Announce Type: cross Abstract: Speech-to-speech dialogue models increasingly support persona control, yet existing spoken role-playing benchmarks remain largely character-centric an
arXiv:2609.16878v1 Announce Type: cross Abstract: Despite its crucial role in video object removal (VOR), existing evaluation paradigms face two critical limitations: questionable references and a mis
I reallise this is not exactly a locally hostable model, at least definitly not on my hardware. But I figured some of you might have some insight on this. I run an agentic coding benchmark. The model
arXiv:2510.18368v2 Announce Type: replace Abstract: We present extbf{Korean SimpleQA (KoSimpleQA)}, a benchmark for evaluating factuality in large language models (LLMs) with a focus on Korean cultura
arXiv:2601.15645v2 Announce Type: replace Abstract: Large-scale language models (LLMs) often offer clinical judgments based on incomplete information, increasing the risk of misdiagnosis. Existing stu
arXiv:2510.13852v3 Announce Type: replace-cross Abstract: Is an LLM telling you different facts than it's telling me? This paper introduces ConsistencyAI, an independent benchmark for measuring the fa
arXiv:2609.08149v1 Announce Type: new Abstract: SWE-Bench Pro has emerged as a standard benchmark for evaluating software engineering agents on challenging repository-level tasks. However, our analysi
arXiv:2609.05902v1 Announce Type: new Abstract: Recent image generation systems increasingly combine multimodal understanding, reasoning, and synthesis, suggesting that they may do more than render pl
arXiv:2602.00288v4 Announce Type: replace Abstract: Fine-grained spatio-temporal understanding is essential for video reasoning and embodied AI. Yet, while Multimodal Large Language Models (MLLMs) mas
arXiv:2609.06245v1 Announce Type: new Abstract: Multimodal Large Language Models (MLLMs) perform strongly on general visual understanding tasks such as visual question answering, yet they often strugg
We're introducing Q2D-Web (Query2Doc-Web), a benchmark and public leaderboard for evaluating retrieval in agentic RAG systems. Q2D-Web tests how embedding models perform on large-scale web search usin
My XFX Radeon RX 7900 GRE 16GB Vram GPU struggles with models over 20B size. I added my Radeon RX 480 8GB Vram GPU to the system and ran a few benchmarks using llama.cpp Ubuntu Vulkan build 10453. htt
arXiv:2609.03597v1 Announce Type: new Abstract: Land ownership in Bangladesh is recorded in Ana-Ganda-Kora-Kranti-Til, a base-16 positional fraction system with dedicated Unicode glyphs, no mainstream
arXiv:2609.02339v1 Announce Type: cross Abstract: World modeling requires a predictive model to maintain and update an internal state adequate for reasoning about the consequences of actions. We intro
arXiv:2609.01982v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used as assistants for statistical and data science work, yet existing evaluations largely assume the anal
arXiv:2609.01691v1 Announce Type: new Abstract: Vision-language models (VLMs) are increasingly used to make decisions from visual inputs. We introduce FAIRLENS, a benchmark and evaluation framework fo
arXiv:2609.00048v1 Announce Type: cross Abstract: GUI world models are increasingly evaluated as one-step next-screen predictors, yet their intended use is often as multi-step environments for GUI age
arXiv:2609.00062v1 Announce Type: cross Abstract: Data contamination undermines the reliable evaluation of large language models (LLMs) on mathematical problem solving. While rewriting-based evaluatio
arXiv:2609.00330v1 Announce Type: cross Abstract: In contact centers, real-time agent-assist tools determine, for each of many predefined topics, whether a live customer utterance is relevant and disp
arXiv:2505.13180v3 Announce Type: replace Abstract: Integrating Large Language Models with symbolic planners is a promising direction for obtaining verifiable and grounded plans, with recent works ext
arXiv:2609.01056v1 Announce Type: new Abstract: Despite the growing use of LLM-powered agents to solve multi-step tasks in complex environments, existing benchmarks rarely test state preservation, per
arXiv:2608.30188v1 Announce Type: new Abstract: Large Language Models (LLMs) show great potential as clinical agents, yet existing benchmarks reduce clinical workflows to static predictions or unconst
arXiv:2608.28762v1 Announce Type: new Abstract: Recent advances in visual question answering (VQA) and multimodal large language models (MLLMs) have enabled natural-language reasoning over traffic sce
arXiv:2608.30517v1 Announce Type: new Abstract: Benchmark saturation and data contamination increasingly obscure genuine scientific reasoning in frontier LLMs. We introduce extsc{ScienceArena}, an oly
arXiv:2608.30716v1 Announce Type: new Abstract: Recent advances in Large Multimodal Models (LMMs) have greatly improved video understanding, yet their ability to reason about human-centered social sit
arXiv:2608.31108v1 Announce Type: new Abstract: Efficient evaluation changes the protocol used to support claims about model behavior, yet it is rarely tested whether those claims remain stable after
arXiv:2608.28641v1 Announce Type: cross Abstract: Most evaluations for coding agents are conducted exclusively in English, which does not reflect real-world multilingual deployment. We present Termina
arXiv:2608.29577v1 Announce Type: new Abstract: Traditional video highlight detection relies on a narrow, event-centric definition of saliency, which often fails to generalize to unconstrained persona