Search: “evaluation”

I am very proud of our team for releasing EDINET-Bench, and it is fantastic to see a Japanese financial dataset recognized at #ICLR2026 this week. We need more diverse, non-English datasets to evaluat

This post draws a humorous parallel between using images of salarymen eating ramen as a test case for AI image generation models in Eastern contexts and the Western 'Will Smith eating spaghetti' meme,

GDPval is one of the most important benchmarks of AI ability because it is based on human expertise. It compares expert human performance to AI performance using expert human judges who spend an avera

Data Driven Agent Design with Evals & Hill Climbing Algorithms this is a mental model dump i’ve been thinking through + iterating on as we’re building self-improvement infra around agents: - mining Tr

Simon Willison discusses results from a 'flamingo on a unicycle' test on his blog, likely comparing AI model performance including Qwen. The post appears to reference previously undisclosed or unconve

NEW Research from Google. Integration test failures are painful because the signal is buried in messy logs. Massive output, heterogeneous systems, low signal-to-noise ratio, and unclear root causes. T

Today we’re announcing Ternary Bonsai: Top intelligence at 1.58 bits Using ternary weights {-1, 0, +1}, we built a family of models that are 9x smaller than their 16-bit counterparts while outperformi

AI21 Labs shared findings indicating that their model does not simply memorize solutions but instead identifies and recognizes aesthetic qualities associated with high-quality outputs, such as minimal

Document OCR benchmarks are still an open problem Existing document OCR benchmarks are either too narrowly focused on a specific type (e.g. FinTabNet, ChartQA), or on documents that aren’t reflective

It took me a few hours yesterday, but I got access to the case files for Olaolukitan Adon Abel previous se*ual battery case from Savannah GA that caused him to be BANISHED from the ENTIRE CITY. I'll d

Parsing complex tables in PDFs is extremely challenging. Existing metrics for measuring table accuracy, like TEDS (tree edit distance similarity), overweight exact table structure and underweight sema

Today we launched Gemini 3.1 Flash TTS, our most expressive and controllable text-to-speech model yet. This launch [excitement] includes audio tags! 🗣🏷 Audio tags [explanatory] are a seamless way to g

As of April 2026, Latent Space published a curated ranking of the top locally-runnable AI models, providing a research-based overview of the best open-weight or self-hostable models available at that

Ethan Mollick discusses the concept of 'emergence' in large language models (LLMs), referring to the phenomenon where AI systems appear to suddenly develop unexpected capabilities as they scale. The p

Everyone who think vision is solved should look at some enterprise documents We’re open sourcing the first document OCR benchmark for the agentic era, ParseBench. Document parsing is the foundation of

Ethan Mollick discusses the economic and financial risks surrounding data center construction, distinguishing between the risks inherent in financing methods and the broader arguments driving investme

The model is not the agent. The harness is. You need to read this recent study, and a blog post from @hwchase17 ... (links below). It will resonate deeply. This diagram from a recent paper captures so

We’re open sourcing the first document OCR benchmark for the agentic era, ParseBench. Document parsing is the foundation of every AI agent that works with real-world files. ParseBench is a benchmark t

Your AI isn't failing because of 'bad chunking.' 🛑 It’s failing because of a vibe mentality. Our Head of DevRel @RoieSchwabco explains why anecdotal testing is the #1 roadblock for AI devs: ✅ Determin

Harrison Chase, co-founder of LangChain, argues that while prompt engineering is foundational and can get you 80% of the way to a functional AI system, the remaining 20% of engineering effort involves

Ethan Mollick shares an assessment of an AI model (likely a newer or smaller model) that, while not matching the performance of the leading frontier models (likely referring to top offerings from Open

MiniMax M2.7 is now available on Together AI, offering a powerful open-source model trained using a self-directed reinforcement learning loop. This training methodology enabled the model to achieve th

NASA's Artemis II mission concluded successfully on April 10–11, 2026, when the Orion spacecraft (named *Integrity*) splashed down in the Pacific Ocean off the coast of San Diego at 8:07 p.m. EDT (...

a useful mental model on how teams can think about good data design to improve their models/agents: Evals ~= Training Data ~= Environments - in Classical Deep Learning, we learn from each training exa

DeCarlos Brown Jr., a man diagnosed with schizophrenia who had been previously arrested 14 times on charges ranging from illegal firearm possession to felony robbery and assault, stabbed Ukrainian ...

This 10-min read from @Vtrivedy10 changes how you build AI agents. Most people are stuck in the same loop; switching models when agents break. The reframe: evals are the training data for your harness

Cursor's code review agent, **Bugbot**, now supports real-time self-improvement by learning from activity on pull requests. Bugbot reviews hundreds of thousands of PRs per day and uses signals fro...

GLM-5.1 is Zhipu AI's next-generation flagship model for agentic engineering, achieving state-of-the-art performance on SWE-Bench Pro and leading its predecessor GLM-5 by a wide margin on NL2Repo (...

Visually rich documents are especially challenging for agents. Tables, charts, and images often break traditional document pipelines, making complex reasoning difficult📄 So we teamed up with @lancedb