Knowledge catalogue
arXiv:2610.19240v1 Announce Type: new Abstract: Financial disclosures express the underlying relationships of double-entry accounting through unstructured text. While financial LLM benchmarks predomin
arXiv:2610.19165v1 Announce Type: cross Abstract: Many real-world questions demand not a single fact but an organized collection of them, yet existing factual knowledge benchmarks almost exclusively t
arXiv:2505.13546v2 Announce Type: replace-cross Abstract: Automatic prompt generation plays a crucial role in enabling general-purpose multi-agent systems to perform diverse tasks autonomously. Existi
arXiv:2610.17665v2 Announce Type: new Abstract: Can vision-language models turn 2D observations into 3D understanding that supports precise action? This requires inferring object geometry, anticipatin
arXiv:2610.17326v1 Announce Type: new Abstract: Household robots must successfully navigate im-perfect circumstances, both in their internal models and the external world. We introduce a benchmark of
arXiv:2610.17898v1 Announce Type: new Abstract: Sparse autoencoders decompose language model representations into interpretable features, but feature interpretation in this literature remains predomin
arXiv:2610.18098v1 Announce Type: new Abstract: Deleting singular components offers a precise way to remove a capability from a language model, yet the first-order spectral saliency used to rank compo
arXiv:2610.14397v1 Announce Type: new Abstract: Augmenting worker capabilities in the physical world through guidance has been a decades-long goal for AI agents. One key supporting capability is to un
arXiv:2610.17531v1 Announce Type: new Abstract: While it is known that quantization of large language models (LLMs) reduces memory usage via lower-bit weights, studies quantifying the resulting impact
🔎 Announced at Interrupt: AI Search in LangSmith You used to need an evaluator or read hours of conversations to find traces that answer questions such as: 💬 Does the user seemed frustrated?' 💬 Where
New paper from Microsoft and colleagues on generating enterprise data for agents. If you train or evaluate agents on long-running business work, it's worth your time. (bookmark it) MiniCorp is an offi
arXiv:2601.18901v2 Announce Type: replace Abstract: Knowledge probing quantifies how much relational knowledge a language model (LM) has acquired during pre-training. Existing knowledge probes evaluat
arXiv:2610.11050v1 Announce Type: new Abstract: Computer-use agents are capable of completing complex tasks, increasing the use of automatic judges to determine success, either for training or for eva
arXiv:2610.10946v1 Announce Type: new Abstract: Large language models (LLMs) are entering wildfire management, where overstated evaluations can cost property and lives. How do they perform on wildfire
arXiv:2610.11069v1 Announce Type: new Abstract: Large language model (LLM) assistants are being deployed to clinicians across health systems, and judgments about their readiness rest largely on benchm
arXiv:2610.10942v1 Announce Type: new Abstract: Reinforcement learning environments are now a primary lever for improving large language model (LLM) capabilities in post-training, yet most agentic ben
arXiv:2610.09272v1 Announce Type: new Abstract: Attention routing requires a signal that predicts the value of attention on the current prefix. We evaluate whether hidden-state extrapolation error, cu
arXiv:2510.13276v2 Announce Type: replace Abstract: The rapid advancement of long-context vision language models (LCVLMs) has led to a significant expansion of their context windows. However, an exten
arXiv:2610.02549v1 Announce Type: new Abstract: Time and event expression extraction are fundamental temporal reasoning tasks, but the problem remains difficult due to annotation ambiguity, domain sen
arXiv:2505.21870v2 Announce Type: replace-cross Abstract: Retrieval-augmented generation (RAG) generally enhances large language models' (LLMs) ability to solve knowledge-intensive tasks. But RAG coul
AI researchers Zainhas and Parthsareen from Ollama delivered a breakout presentation on October 2, 2026, focusing on evaluating open-source models under real‑world production traffic rather than just
arXiv:2610.00649v1 Announce Type: cross Abstract: Synthetic speech detection is critical for audio security, but performance can degrade when labeled data are scarce and evaluation conditions differ f
arXiv:2510.23191v2 Announce Type: replace Abstract: Predictive benchmarking, evaluating machine learning models based on predictive performance and competitive ranking, is central to machine learning
arXiv:2609.38260v1 Announce Type: cross Abstract: Values such as honesty, autonomy, and confidentiality are often regarded as general principles underpinning AI alignment. However, what it means to ac
arXiv:2603.01209v3 Announce Type: replace Abstract: In CodeAct, language-model agents write Python that calls tools and use execution feedback to choose actions. Persistent runtimes preserve Python va
arXiv:2609.38797v1 Announce Type: cross Abstract: As AI systems move from static repositories to agents that are capable of continual adaptation and learning, maintaining their trustworthiness means e
arXiv:2609.40198v1 Announce Type: cross Abstract: Speech-to-speech systems must solve tasks whose requirements emerge across conversational turns. We introduce SpeechConversationBench (SCB), a focused
arXiv:2609.37501v1 Announce Type: cross Abstract: We propose RegLLM, a diagnostic harness for bounded autonomy in regulated agentic workflows. It instruments six trustworthiness signals: citation vali
arXiv:2602.18998v2 Announce Type: replace Abstract: LLM agents are increasingly expected to operate as general-purpose systems that resolve real-world user requests, yet their dynamic scaling behavior
arXiv:2609.37143v1 Announce Type: cross Abstract: Modern coding agents can deliver increasingly large repository-level changes, and recent benchmarks reflect this by emphasizing long-horizon tasks wit
arXiv:2609.36253v1 Announce Type: cross Abstract: Large language models (LLMs) show considerable potential in simulating human attitudes and preferences. Prior work finds that LLM-generated responses
arXiv:2609.35900v1 Announce Type: cross Abstract: The integration of large language models (LLMs) into scientific workflows is accelerating, yet their ability to reconstruct the reasoning underlying p
arXiv:2609.31675v1 Announce Type: new Abstract: We introduce the Active Causal Discovery Benchmark (ACDB), an SCM-grounded environment for evaluating whether LLM agents recover causal graph structure
arXiv:2609.32157v1 Announce Type: new Abstract: Vision-Language-Action (VLA) models for autonomous driving produce natural-language reasoning alongside predicted trajectories, but whether this reasoni
arXiv:2609.33807v1 Announce Type: new Abstract: How well can general-purpose multimodal models turn visual understanding and reasoning into embodied manipulation via executable code? We introduce Code
arXiv:2609.32696v1 Announce Type: cross Abstract: Data shape determines how features are structured, how patterns are separated, and how distributions cover the underlying domain. Poor data shape can
arXiv:2609.34215v1 Announce Type: cross Abstract: Evaluating how LLM agents recover from mid-task failures is central to deploying reliable agentic systems. Existing checkpoint-based benchmarks measur
Vultr is evaluating NVIDIA’s Agent Safety Platform as enterprise agents move from experimentation to production and receive real autonomy over internal systems. The platform addresses trust by securin
arXiv:2609.30981v1 Announce Type: new Abstract: Reliable online video question answering requires tracking state transitions while selectively abstaining when visual evidence is insufficient. Existing
We comprehensively evaluated 16 recent frontier VLMs - including Opus 5.5 and GPT-6 Sol/Luna - on whether higher effort led to better document parsing performance. Higher effort typically leads to imp
In September 2026, Sebastian Thrun announced **PhilosophyBench**, the first independent, large‑scale benchmark created by Stanford’s AI Lab and HCI group to evaluate artificial intelligence’s philosop
arXiv:2512.04125v2 Announce Type: replace Abstract: Large language models (LLMs) have demonstrated several emergent behaviors with scale, including reasoning and fluency in long-form text generation.
arXiv:2609.27607v1 Announce Type: cross Abstract: An AI-generated radiology report can resemble a physician's report while omitting an abnormality, adding an unsupported finding, or reversing its pres
arXiv:2609.27678v1 Announce Type: new Abstract: Contract inference requires multiple judgments about a shared document, but aggregate accuracy can conceal changes in the individual decisions. Repeated
arXiv:2609.23806v1 Announce Type: new Abstract: Many knowledge-work benchmarks are constructed around individual tasks, with the context needed for each task selected together with or after the task h
arXiv:2609.23980v1 Announce Type: cross Abstract: AI agents now report vulnerabilities faster than maintainers can review them. Reports often depend on security properties specific to the application,
arXiv:2609.23784v1 Announce Type: new Abstract: Robotic bin packing requires long-horizon sequential decision-making, as each object placement affects the available space for subsequent packing. Exist
arXiv:2609.22281v1 Announce Type: new Abstract: Foundation models have recently demonstrated strong capabilities across a wide range of medical imaging tasks. However, their performance in structured
arXiv:2609.21386v1 Announce Type: cross Abstract: Comprehensive video understanding is crucial for advancing artificial intelligence toward the intricate dynamics of the physical world. While recent a
A new release of our Agent Skills and Docs eval library, Cultivar is available! Now, use @typesafeai's Jev model to evaluate how well your agents use docs and Skill against specified tasks in Modal sa
arXiv:2609.20152v1 Announce Type: new Abstract: Generally, most voice agents are cascaded systems, i.e., an ASR model transcribes the caller's audio, a language model reads the transcript and decides
The simple truth: not even Anthropic has a plan for how to make AI safe in advance. There is no theory. I've asked! Outside evaluators don't change the situation. No one at these companies has workabl
🔥 take: @CompleteSkeptic//TypeSafe killed Quantum Computing Jev feels a bit like bringing the quantum-computing intuition to AI: evaluate a distribution of possibilities in parallel instead of generat
arXiv:2609.16093v1 Announce Type: new Abstract: A shopping conversation has many routes to the same cart, and a task-success rate reduces all of them to one score. We build a deterministic and reprodu
Cristina Criddle / Financial Times: Sources: OpenAI and Anthropic staff felt blindsided by Dario Amodei's and Sam Altman's calls to slow the frontier; some fear evaluators may compromise security — St
arXiv:2609.12839v2 Announce Type: replace-cross Abstract: The proliferation of highly capable open-weight Small Language Models (SLMs) democratizes access to advanced cybersecurity capabilities, posin
arXiv:2609.13192v1 Announce Type: new Abstract: This study compares traditional machine learning models and Large Language Model (LLM)-generated rule-based systems for heart disease prediction using t
arXiv:2512.24565v4 Announce Type: replace Abstract: Large Language Models (LLMs) are increasingly serving as autonomous agents, and their utilization of external tools via the Model Context Protocol (
arXiv:2609.15683v1 Announce Type: new Abstract: While In-Context Learning (ICL) enables models to adapt from exemplars without parameter updates, multimodal ICL remains largely underexplored, particul
Joseph Cox / 404 Media: Internal OpenAI docs detail contractors evaluating anonymized prompts and chats to improve the models; model training is turned on by default for consumer plans — Humans are re