Search: “evaluation”

I really do think we'll see a lot of value accrue in AI startups building 'model routing as a service' Not just OpenRouter - this includes a much broader set of verticalized agents and infrastructure.

Code volume is not a reliable metric for measuring developer productivity, as writing more lines of code does not necessarily correlate with delivering more value or solving problems more effectively.

Political appointees vetting science funding? Trump is doing to science what the Right accused previous administrations of doing. But in previous administrations, there was no political control of res

// The Meta-Agent Challenge // How good are current agents at self-improving? This is a great paper covering some of the challenges. They propose the Meta-Agent Challenge (MAC), where they give a codi

Want to hear about where ASR is headed? Check out @HuggingFace's webinar about their new Far-Field ASR Leaderboard with @shinjiw_at_cmu and our own @Julianfmack on Thursday, June 11th. We'll be chatti

We've made a breakthrough in self-evolving AI scientists moving from 'search' to 'principled discovery': Scientific discovery requires that the search space itself changes, and an AI scientist must pe

Gary Marcus discusses investment strategy between potential OpenAI and Anthropic IPOs, likely weighing factors such as the companies' technological capabilities, market positioning, business models, a

Devin is an AI software engineer product that likely includes a performance guarantee and specific eligibility requirements for users, with details available on Cognition AI's official guarantee page.

Our team is at CVPR 2026 if you want to come say hi :) We're presenting ParseBench at CVPR 2026! ParseBench is the most comprehensive document understanding benchmark for VLMs. ✅ It contains 2k pages

We're presenting ParseBench at CVPR 2026! ParseBench is the most comprehensive document understanding benchmark for VLMs. ✅ It contains 2k pages of real-world enterprise documents ✅ It has comprehensi

This post likely discusses an innovative reward function design that optimizes for reasoning efficiency in AI systems, possibly in the context of language models or reinforcement learning. The entry a

Thanks @hwchase17 @bryonkuchML for the inspiration and the nudge from @MEGAcodePaul - took your feedback seriously. I have developed a boilerplate that covers the full optimization loop that productio

We wrapped a live session on M3 yesterday with the @togethercompute team & our researchers @zpysky1125 and @HaohaiSun A few highlights 🧵 1. MSA (MiniMax Sparse Attention) is the star ⭐️. Unlike CSA/HC

Big day for American open models... Nemotron 3 Ultra is now the strongest US open-weight model tested, while apparently serving 300+ tok/s 🤯 Comparable large DeepSeek/Kimi models are usually 50-100 to

MiniMax M3, an upcoming AI model, is expected to be released soon and will undergo comprehensive testing within a custom coding agent framework. A detailed technical review of the model's performance

Agree! was talking about this with @havoyan just a few days ago. That's also the reason why so much of the value has been accruing to the frontier models in my opinion (cc @GavinSBaker) because if you

Had a blast yesterday attending at @techeurope_'s Applied AI Conference in Berlin! I had a talk about building document agents and agentic development in general, that you can find here: https://astra

Having Grok Build sub-agents to iterate several ideas for me on data loading, batching, inference, and writing results to files for dense datasets before I went to sleep. It gave me a nice summary of

It’s a good day when the Pope vouches for your recent comment in Nature. The Pope is making exactly our point. LLMs “may imitate or even simulate, but they do not understand.” This is the core epistem

Production traffic from frontier models is a golden data asset. If you can efficiently mine the traces, filter for quality, and fine-tune smaller models on them, you get specialized performance at a f

Tip on Grok Build + Grok 4.3 VLM One of my critical tasks is to keep Grok VLM in the loop. Throwing a default system prompt usually yields poor results due to lack of context. Here is how to scale: -

As AI agents handle increasingly complex tasks with extended context windows, the cost structure of LLM usage has shifted such that input tokens now represent the majority of price-equivalent expenses

Governance should lead to more building, not less Agent development is so much more fun when people don't need to ration tokens because one runaway agent might blow up the bill Understand spend, cap t

📢Qwen3.7-Max just hit #3 on ITbench-AA — a fresh benchmark testing how well models handle real-world enterprise IT tasks, agentic-style. 🔧Agentic era, go with Qwen.🏃🏃 Artificial Analysis and IBM Resea

completing the loop is paramount most unrecognized step in the development lifecycle is iteration on the process itself everyone is talking about self-optimizing loops in software & agents. but what d

Cost controls are one of the main benefits of langsmith LLM gateway We’re seeing in a bunch of use cases adoption is getting to a point where cost really starts to matter Your agents can burn through

ESMFold2 is described as a high-performance protein structure prediction model that demonstrates particular strength in predicting antibody-antigen complex structures, reportedly outperforming AlphaFo

The MiniMax M2 series was one of the most widely used open-weight LLM series earlier this year. Now, we got a technical report with some interesting tidbits. I summarized some of them below: 1. Full a

langsmith engine is an agent that helps you spin the optimization loop for your own agents everyone is talking about self-optimizing loops in software & agents. but what does that actually mean? in my

System scaling is the next real bottleneck in agentic AI. If you build agent orchestration layers, this is a clean map of where the engineering leverage actually sits. The labs own the model. You own

Trace data is literally worth its weight in gold these days, if you know what to do with it! As has been established, creating effective agents requires shipping early, observing behavior, and iterati

Its very limiting that a big set of very hard problems that we have just lying around are Erdos problems. Don’t get me wrong, they are quite cool, but we really need hard problems repositories for man

Gary Marcus examines and fact-checks recent claims made by OpenAI and Anthropic in their public announcements and media coverage. The post links to detailed analysis questioning the mathematical valid

The post discusses how AI can be effectively used for peer review processes when combined with human reviewers, while noting that AI's capabilities in this area continue to improve. It implies that hy

⚠️👇 🚨Breaking ⚠️ If we can’t make AI agents follow rules, we are screwed. New study from METR reports that “when the agents were faced with hard tasks, they routinely violated constraints” This—routin

Yohei Nakajima requests a critical analysis of the ActiveGraph AI documentation, examining its approach and positioning within the evolving landscape of AI technologies. The post appears to seek detai

Patches represent a mechanism for establishing trust boundaries in systems by decoupling the intent to change state from the acceptance and policy-driven implementation of that change. This separation

Harrison Chase provides a detailed technical overview of the architecture and development process behind LangSmith Engine, sharing key learnings and practical tips accumulated during its construction.

this has probably been the most interesting project i've ever worked on. we created the best agent at finding problems with your agent. and we integrated it with the best observability platform to spi

LLM apps fail in ways normal logs can’t explain. Same input, different outputs. Subtle drift. Silent hallucinations. That’s the problem space LangSmith by @LangChain was built for: observability + eva

The U.S. faces a chaotic #AI regulatory patchwork with over 1,200 state bills introduced in 2025 and no unified federal framework, researchers @JeffSonnenfeld and Stephen Henriques of @YaleSOM and @Ga

Gary Marcus draws a historical parallel between Bernie Madoff's claimed 16.3% annual returns (which proved to be a Ponzi scheme) and Sam Altman's promised 17.5% returns, warning that similarly implaus

Many AI agents in finance rely on extremely high quality context engineering from documents 📑 They can be roughly divided into two categories: 1️⃣ Repetitive, operational work common in back-office us

arXiv:2605.14855v1 Announce Type: cross Abstract: Forecasting within signal processing pipelines is crucial for mitigating delays, particularly in predicting the dynamic movements of objects such as N

big launches today! 🚀Launching: LangSmith Engine LangSmith Engine is an agent that sits on top of your traces It runs in the background and automatically identifies issues It then proactively suggests

SWE-ZERO and AgentTrove are trending datasets on Hugging Face related to software engineering and agentic AI systems. The post indicates growing interest in trace datasets that capture agent behaviors