Search: “gemini”

Measuring Five-Nines Reliability: Sample-Efficient LLM Evaluation in Saturated Benchmarks

model-releasesarxiv-cs-lg

arXiv:2605.11209v1 Announce Type: new Abstract: While existing benchmarks demonstrate the near-perfect performance of large language models (LLMs) on various tasks, this apparent saturation often obsc

13 May 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Three Regimes of Context-Parametric Conflict: A Predictive Framework and Empirical Validation

DGX agent

arXiv:2605.11574v1 Announce Type: new Abstract: The literature on how large language models handle conflict between their training knowledge and a contradicting document presents a persistent empirica

model-releasesarxiv-cs-cl

13 May 2026

AgentCollabBench: Diagnosing When Good Agents Make Bad Collaborators

arXiv:2605.08647v1 Announce Type: cross Abstract: Multi-agent systems achieve state-of-the-art outcomes through peer collaboration. However, when an agent in the pipeline silently drops a constraint,

Are vision-language models ready to zero-shot replace supervised classification models in agriculture?

arXiv:2512.15977v3 Announce Type: replace Abstract: Vision-language models (VLMs) are increasingly proposed as general-purpose solutions for visual recognition tasks, yet their reliability for agricul

Beyond Accuracy: Evaluating Strategy Diversity in LLM Mathematical Reasoning

arXiv:2605.09292v1 Announce Type: new Abstract: Large language models now achieve high final-answer accuracy on mathematical reasoning benchmarks, but accuracy alone does not capture reasoning flexibi

Byte-Exact Deduplication in Retrieval-Augmented Generation: A Three-Regime Empirical Analysis Across Public Benchmarks

arXiv:2605.09611v1 Announce Type: new Abstract: This preprint presents an empirical analysis of byte-exact chunk-level deduplication in Retrieval-Augmented Generation (RAG) pipelines. We measure conte

ChatbotManip: A Dataset to Facilitate Evaluation and Oversight of Manipulative Chatbot Behaviour

arXiv:2506.12090v2 Announce Type: replace Abstract: This paper introduces ChatbotManip, a novel dataset for studying manipulation in Chatbots. It contains simulated generated conversations between a c

Continual Harness: Online Adaptation for Self-Improving Foundation Agents

arXiv:2605.09998v1 Announce Type: cross Abstract: Coding harnesses such as Claude Code and OpenHands wrap foundation models with tools, memory, and planning, but no equivalent exists for embodied agen

GUARD: Guideline Upholding Test through Adaptive Role-play and Jailbreak Diagnostics for LLMs

arXiv:2508.20325v3 Announce Type: replace-cross Abstract: As Large Language Models (LLMs) become increasingly integral to various domains, their potential to generate harmful responses has prompted si

K12-KGraph: A Curriculum-Aligned Knowledge Graph for Benchmarking and Training Educational LLMs

arXiv:2605.09635v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used in K-12 education, yet existing benchmarks such as C-Eval, CMMLU, GaokaoBench, and EduEval mainly eva

Less Diverse, Less Safe: The Indirect But Pervasive Risk of Test-Time Scaling in Large Language Models

arXiv:2510.08592v3 Announce Type: replace-cross Abstract: Test-Time Scaling (TTS) improves LLM reasoning by exploring multiple candidate responses and then operating over this set to find the best out

MaD Physics: Evaluating information seeking under constraints in physical environments

arXiv:2605.10820v1 Announce Type: new Abstract: Scientific discovery is fundamentally a resource-constrained process that requires navigating complex trade-offs between the quality and quantity of mea

MathConstraint: Automated Generation of Verified Combinatorial Reasoning Instances for LLMs

arXiv:2605.08498v1 Announce Type: cross Abstract: We introduce MathConstraint, a hard, adaptive benchmark for evaluating the combinatorial reasoning capabilities of LLMs. We combine constraint satisfa

MemPrivacy: Privacy-Preserving Personalized Memory Management for Edge-Cloud Agents

arXiv:2605.09530v1 Announce Type: cross Abstract: As LLM-powered agents are increasingly deployed in edge-cloud environments, personalized memory has become a key enabler of long-term adaptation and u

Metal-Sci: A Scientific Compute Benchmark for Evolutionary LLM Kernel Search on Apple Silicon

arXiv:2605.09708v1 Announce Type: cross Abstract: We present Metal-Sci, a 10-task benchmark of scientific Apple Silicon Metal compute kernels spanning six optimization regimes (stencils, all-pairs in

Omni-DeepSearch: A Benchmark for Audio-Driven Omni-Modal Deep Search

model-releasesarxiv-cs-lg

arXiv:2605.08762v1 Announce Type: cross Abstract: Current omni-modal benchmarks mainly evaluate models under settings where multiple modalities are provided simultaneously, while the ability to start

OpenAI contacted me to say “Study Mode is still live and accessible via /study and /learn shortcuts” so that’s good, although the official s…

model-releasesethan-mollick--x

OpenAI contacted me to say “Study Mode is still live and accessible via /study and /learn shortcuts” so that’s good, although the official study mode page doesn’t mention that. (I don’t think slash co

Overconfident and Blind to Details: Fixing Prompt Insensitivity with Abductive Preference Learning

arXiv:2510.09887v2 Announce Type: replace Abstract: Vision and language models frequently ignore semantically critical input edits, defaulting to pretraining priors. For example, models will confident

PhyGround: Benchmarking Physical Reasoning in Generative World Models

arXiv:2605.10806v1 Announce Type: cross Abstract: Generative world models are increasingly used for video generation, where learned simulators are expected to capture the physical rules that govern re

Polymath: A Challenging Multi-modal Mathematical Reasoning Benchmark

arXiv:2410.14702v2 Announce Type: replace Abstract: Multi-modal Large Language Models (MLLMs) exhibit impressive problem-solving abilities in various domains, but their visual comprehension and abstra

Process Matters more than Output for Distinguishing Humans from Machines

arXiv:2605.06524v2 Announce Type: replace Abstract: Reliable human-machine discrimination is becoming increasingly important as large language models and autonomous agents are deployed in online setti

Soohak: A Mathematician-Curated Benchmark for Evaluating Research-level Math Capabilities of LLMs

arXiv:2605.09063v1 Announce Type: new Abstract: Following the recent achievement of gold-medal performance on the IMO by frontier LLMs, the community is searching for the next meaningful and challengi

The Extrapolation Cliff in On-Policy Distillation of Near-Deterministic Structured Outputs

arXiv:2605.08737v1 Announce Type: cross Abstract: On-policy distillation (OPD) is widely used for LLM post-training. When pushed with a reward-extrapolation coefficient lambda > 1, the student can lif

The Metacognitive Probe: Five Behavioural Calibration Diagnostics for LLMs

arXiv:2605.09844v1 Announce Type: new Abstract: The Metacognitive Probe is an exploratory five-task, 15-slot diagnostic that decomposes an LLM's confidence behaviour into five behaviourally-distinct d

To Redact, or not to Redact? A Local LLM Approach to Deliberative Process Privilege Classification

arXiv:2605.10211v1 Announce Type: cross Abstract: Government transparency laws, like the Freedom of Information (FOIA) acts in the United States and United Kingdom, and the Woo (Open Government Act) i

Towards Conversational Medical AI with Eyes, Ears and a Voice

arXiv:2605.09272v1 Announce Type: new Abstract: The practice of medicine relies not only upon skillful dialogue but also on the nuanced exchange and interpretation of rich auditory and visual cues bet

Towards On-Policy Data Evolution for Visual-Native Multimodal Deep Search Agents

arXiv:2605.10832v1 Announce Type: new Abstract: Multimodal deep search requires an agent to solve open-world problems by chaining search, tool use, and visual reasoning over evolving textual and visua

Try Grok Voice

model-releaseselon-musk--x

Try Grok Voice Grok Voice Think Fast 1.0 ranks #1 on the Artificial Analysis τ-Voice benchmark for real-world agentic customer service resolution Absolutely outperforming GPT-Realtime-2 (High) and Gem

UTS at PsyDefDetect: Multi-Agent Councils and Absence-Based Reasoning for Defense Mechanism Classification

arXiv:2605.09769v1 Announce Type: new Abstract: This paper describes our system for classifying psychological defense mechanisms in emotional support dialogues using the Defense Mechanism Rating Scale

VidNum-1.4K: A Comprehensive Benchmark for Video-based Numerical Reasoning

arXiv:2604.03701v2 Announce Type: replace Abstract: Video-based numerical reasoning provides a premier arena for testing whether Vision-Language Models (VLMs) truly 'understand' real-world dynamics, a

Architecting a resilient, scalable and secure foundation for the agentic era

model-releasesgoogle-cloud-ai

Across the public sector, the conversation has shifted; we are no longer just talking about the potential of AI, we are already seeing the impact. While visionary leadership and cultural buy-in are cr

Cloud Storage Rapid: Turbocharged object storage for AI and analytics

model-releasesgoogle-cloud-ai

At Google Cloud Next ’26 we announced Cloud Storage Rapid, a family of object storage capabilities for data-intensive workloads like AI and analytics. Out of the gate, Cloud Storage Rapid consists of

Cluster-level reliability for trillion-parameter models on TPUs

model-releasesgoogle-cloud-ai

Frontier AI models have redefined the unit of compute. At trillion-parameter scale, AI training requires thousands of interconnected components, orchestrated in industrial-scale deployments to operate

Domain-level metacognitive monitoring in frontier LLMs: A 33-model atlas

arXiv:2605.06673v1 Announce Type: cross Abstract: Aggregate metacognitive quality scores mask within-model variation across MMLU benchmark domains. We administered 1,500 MMLU items (250 per domain, un

End-to-end PDDL Planning with Hardcoded and Dynamic Agents

arXiv:2512.09629v2 Announce Type: replace Abstract: We present an end-to-end framework for planning supported by verifiers. An orchestrator receives a human specification written in natural language a

Fine-tuning a vision-language model for fracture-surface morphology recognition

arXiv:2605.07145v1 Announce Type: cross Abstract: Vision-language models (VLMs) have shown strong potential for scientific image understanding, but general-purpose models often lack the domain-specifi

How Far Are VLMs from Privacy Awareness in the Physical World? An Empirical Study

arXiv:2605.05340v2 Announce Type: replace-cross Abstract: As Vision-Language Models (VLMs) are increasingly deployed as autonomous cognitive cores for embodied assistants, evaluating their privacy awa

IntentGrasp: A Comprehensive Benchmark for Intent Understanding

arXiv:2605.06832v1 Announce Type: cross Abstract: Accurately understanding the intent behind speech, conversation, and writing is crucial to the development of helpful Large Language Model (LLM) assis

MAVEN: Multi-Agent Verification-Elaboration Network with In-Step Epistemic Auditing

arXiv:2605.07646v1 Announce Type: cross Abstract: While explicit reasoning trajectories enhance model interpretability, existing paradigms often rely on monolithic chains that lack intermediate verifi

MobileDev-Bench: A Benchmark for Issue Resolution in Mobile Application Development

model-releasesarxiv-cs-lg

arXiv:2603.24946v2 Announce Type: replace-cross Abstract: Large language models (LLMs) have shown strong performance on automated software engineering tasks, yet existing benchmarks focus primarily on

MultiSoc-4D: A Benchmark for Diagnosing Instruction-Induced Label Collapse in Closed-Set LLM Annotation of Bengali Social Media

arXiv:2605.06940v1 Announce Type: new Abstract: Annotation automation via Large Language Models (LLMs) is the core approach for scaling NLP datasets; however, LLM behavior with respect to closed-set i

NSMQ Riddles: A Benchmark of Scientific and Mathematical Riddles for Quizzing Large Language Models

arXiv:2605.07051v1 Announce Type: new Abstract: Large Language Models (LLMs) have shown good performance on various science educational benchmarks, demonstrating their potential for use in science and

RedDiffuser: Auditing Multimodal Safety Failures in Vision-Language Models via Reinforced Diffusion

arXiv:2503.06223v5 Announce Type: replace Abstract: Large Vision-Language Models (VLMs) are increasingly deployed in open-ended environments, where ensuring reliable safety under multimodal inputs is

SCENE: Recognizing Social Norms and Sanctioning in Group Chats

arXiv:2605.07823v1 Announce Type: new Abstract: Online group chats are social spaces with implicit behavior patterns that, when broken, are often met with social sanctioning from the group. The abilit

SmellBench: Evaluating LLM Agents on Architectural Code Smell Repair

arXiv:2605.07001v1 Announce Type: cross Abstract: Architectural code smells erode software maintainability and are costly to repair manually, yet unlike localized bugs, they require cross-module reaso

Structure Over Scale: Learning Visual Reasoning from Pedagogical Video

arXiv:2601.23251v2 Announce Type: replace Abstract: State-of-the-art vision-language models (VLMs) score impressively on video benchmarks yet stumble on basic visual reasoning tasks involving spatial

The Text Uncanny Valley: Non-Monotonic Performance Degradation in LLM Information Retrieval

arXiv:2605.07186v1 Announce Type: cross Abstract: Existing Large Language Model (LLM) benchmarks primarily focus on syntactically correct inputs, leaving a significant gap in evaluation on imperfect t

TraceAV-Bench: Benchmarking Multi-Hop Trajectory Reasoning over Long Audio-Visual Videos

arXiv:2605.07593v1 Announce Type: new Abstract: Real-world audio-visual understanding requires chaining evidence that is sparse, temporally dispersed, and split across the visual and auditory streams,