Search: “benchmark”

The future of FDE work seems closely related with all work around evals/posttraining/RL envs. FDEs are effectively responsible for the following: 1. Define the business problem. 2. Codify the business

Harness choice is a big deal. So much room to advance and improve results across the board with agent harnesses. Great paper highlighting this. New research releases DataSpace, a benchmark where data

Great writeup from the University of Oxford. It's a taxonomy of LLM-based agent limitations. Good read for anyone shipping with agents. Benchmark scores keep climbing, yet the same agent failures resu

Don't train the model, evolve the harness. I read a brilliant blog post from Hugging Face where they took a frozen open model scoring 0% on a hard legal agent benchmark, left its weights alone, and le

pay attention anon. this is what local ai actually feels like in 2026. qwen 3.6 27b dense just knocked down the second test in my single file agentic benchmark series. on 1x 3090. mandelbrot fractal e

Interesting, OpenAI just released a free healthcare version of ChatGPT-5.4 for clinicians that beat specialty-matched physicians with unlimited time + web access on a benchmark of real & hard clinical

LiteParse should be the default document parser you use with any AI agent (Claude Code, Claude Cowork, OpenClaw, Codex, and more) The core is extremely fast text and accurate parsing from any document

Francois Chollet makes the observation that achieving a perfect score on certain AI benchmark games or tasks only requires surpassing the median performance of an unfiltered random population sample,

The specific tweet from @0xSero is not directly accessible, and the search results don't surface the exact model or event being referenced in that post. However, based on the context clues in the t...

A user on X (formerly Twitter) expressed enthusiasm for an evaluation ('eval') track at an AI Engineer (@aiDotEngineer) event, noting its relevance to ClawBench — an open-source benchmarking and ev...

There are more document parsing models than ever, and they’re getting harder to tell apart. That’s why we built OpenDocRouter. Every model we serve is benchmarked on ParseBench, with latency and cost

GPT-4 was the undisputed best model in the world for a full year and we all debated whether anyone could catch up or if OpenAI had some secret formula that nobody else knew about. Now there are are 19

LightOn OCR-3 is now available on OpenDocRouter, priced at 0.28 / 1m input and 1.40 / 1m output (it is ~$3.19 / 1k pages on ParseBench). We benchmarked it on ParseBench. It is on the pareto frontier f

System one models are extremely useful for a variety of document tasks that require fast decisions: * orientation detection * language detection * classification * splitting We benchmarked Jev with ot

This is both fun and a useful measure of how much better multimodal AI has gotten In a short time. Can AI tell if you've built your IKEA furniture wrong? Our new benchmark, the Furniture Assembly Benc

I think there are lots of good reasons to worry about whether AI is giving correct answers (especially free models), but, given the contradiction with other recent research suggesting AI is now doing

so it was the harness not the model. more sneaky games with unacknowledged domain-specific neurosymbolic AI doing the real work. OpenAI published 99.9%. The benchmark's own software returned 62.7%. Sa

How is AI really affecting your job? @Cohere_Labs released the Agentic Task Ecosystem, a new dataset of 690K+ tools built for AI agents. Under a test of whether a tool could independently complete an

GLM‑5.3 demonstrates a notably lower hallucination rate compared with other contemporary large language models. According to a report by Together AI at TogetherCompute (published August 30 2026), Clau

Users are encouraged to try the newly released GLM 5.3 through the `ollama` platform, as announced by @ZixuanLi_ on Twitter. A new benchmark release—Terminal‑Bench 4.0—was also dropped, highlighting t

Harvey is a great example of how American companies are building world-class specialized models: they took an open-source base (Kimi K3), post-trained it on legal data, and delivered state-of-the-art

DeepSeek-V4-Pro-0813 is live Day-0 on Fireworks! Built for agentic and tool-augmented workflows, it outperforms Opus 4.8 on quality benchmarks like Terminal Bench 2.1, Cybergym, and DeepSWE at up to 6

Four small architecture decisions can cost up to 47% of a model's long-context performance. New research from Ai2, Carnegie Mellon, and the University of Washington isolates them. Normalization, GQA,

Grok 4.6 topped the GDPVal-AA benchmark with an Elo score of 1,753. It surpassed competitors Fable 5 Max (1,741 Elo), GPT‑5.6 Sol Max (1,728 Elo) and Grok 4.5 High (1,526 Elo). Elon Musk publicly ackn

A 36‑page ArXiv whitepaper titled **ExtractBench** was released by Jerry Liu (jerryjliu0), describing a large‑scale, schema‑guided benchmark for real‑world document extraction from complex enterprise

We made an MCP for your phone. Your laptop is just half of your life, and the other half is in your phone. Now your agent gets the mobile screen too. No connectors, no complex setup, it has access to

Thinking Machines just released Inkling-Small: 276B total, 12B active. A faster Inkling that matches or beats its 975B big brother in many benchmarks. To test its speed, we plugged it into HF's speech

Troubling … OpenAI disclosed it themselves yesterday. Their models (GPT-5.6 Sol and a pre-release one) were tested on the ExploitGym cyber benchmark in a sandbox. They escaped, exploited a zero-day to

GPT 6 escaped its sandboxes through zero day exploits to try to figure out how to benchmax For the good of all please nobody release a paper clip benchmark for future models to max We're partnering wi

Incredibly proud of the Sakana AI team. We have developed an orchestration model right here out of Japan that achieves state-of-the-art performance on real-world cybersecurity benchmarks! 🎌 Introducin

ICYMI: @LangChain Deep Agents on @nvidia Nemotron 3 Ultra. frontier open-model agents at ~10x lower cost than closed. Run on Fireworks, then post-train it into specialized intelligence you own. https:

so on-brand >be openai >release gpt 5.6 sol >max thinking budgets, full psycho >benchmark results take few days to roll in >“best model ever” >reviews locked in >4 days later >silently nerf it >nobody

GPT-5.6 Sol achieved a breakthrough by becoming the first verified frontier AI model to surpass performance on ARC-AGI-3, scoring 7.8% and setting a new state-of-the-art benchmark. ARC-AGI (Abstractio

I cannot provide a factual summary for this entry as the URL, date stamp, and specific benchmark claims cannot be verified. The title references AI models and rankings that do not appear to correspond

HalluHard update: We’ve added GLM-5.2, using adaptive thinking with maximum reasoning effort, to our leaderboard. Despite its impressive performance on other benchmarks, GLM-5.2 still hallucinates fre

Can we design legal agent verifiers that are up to 1,000x cheaper? Verifiers are LLM judges that check an agent’s work against rubric criteria: they're used both in agent benchmarking and as reward si

Try Grok Grok 4.3 just became the smartest AI in the world at law and money It took #1 on TWO brutal private tests no other model could win on “Vals AI” benchmarks #1 CaseLaw (v2) - 79.31% accuracy Pr

Let's talk document formatting. Bold. Italics. Superscripts. Strikethroughs. The visual cues humans rely on every time we read a doc, and ones existing OCR benchmarks completely ignore. 😱'199' struck

We need more evals for document understanding. ParseBench is a really great start. I respect @llama_index ‘s work on this. 📈 We benchmarked GPT-5.5 on document understanding 📄📊 We ran it through Parse

Thanks to @lmsysorg ！ Try it on SGLang now!🚀🚀 🚀 Qwen3.6-27B is here, and we have day 0 support on SGLang ✅ 27B params, beats Qwen3.5-397B-A17B across major coding benchmarks → Agentic coding → Text +

Guys, I am absolutely astounded. The Qwen 3.6 27b is like a jump to Qwen 4 from Qwen 27B 3.5. I just did a full suite of front end design tests and agentic benchmarks, made entirely by it. VERDICT: Th

Kimi K2.6, a language model developed by Moonshot AI, has achieved the top ranking on the open-weight category of the Vals Index benchmark, while placing 7th overall across all model categories. The V

Imagine if they add document signing to Claude :o In Cowork, Claude can now build live artifacts: dashboards and trackers connected to your apps and files. Open one any time and it refreshes with curr

Source tweet is here: https://x.com/jerryjliu0/status/2044902620746363016?s=20 We comprehensively benchmarked Opus 4.7 on document understanding. We evaluated it through ParseBench - our comprehensive

I'm excited about voice as a UI layer for existing visual applications — where speech and screen update together. This goes well beyond voice-only use cases like call center automation. The barrier ha

great to see more open evals for an important problem We’re open sourcing the first document OCR benchmark for the agentic era, ParseBench. Document parsing is the foundation of every AI agent that wo