Search: “evaluation”

We've introduced native, agentic spreadsheet extraction 📊 into LlamaParse. Spreadsheets are a wildly different format from PDFs (or any other document format). They can span arbitrarily many rows/colu

BREAKING: GLM-5.3 Flash by @Zai_org debuts at 6th overall on Design Arena with a preliminary Elo of 1343. This places the MIT-licensed, natively multimodal model in the same performance band as GLM-5.

Epic post training work from @isidentical and the @fal team on a strong base from @Hailuo_AI. Strong aesthetic taste and instruction following (#1 on independent benchmarks-artificial analysis and des

GLM-5.3-Flash scores 57 on the Artificial Analysis Intelligence Index. At 0.09 Cost per Task, it sits comfortably on the Intelligence vs. Cost per Task Pareto frontier @Zai_org has released GLM-5.3-Fl

There’s been a lot of debate and speculation about the Hugging Face attack over the last month, and many people have been understandably skeptical of simply taking OpenAI’s word for things. I hope our

Every extraction API demos well on a clean invoice. But what about the scanned form, the nested table, the 40-page financial report with merged headers? We tested 14 frontier systems to find out. Extr

IBM Granite 4.2 is now available on Ollama. 3B, 8B, 30B parameter open models made for enterprise agents. This model is free to use and is licensed for both research and commercial usage. The data cur

If you keep safety rules or coding standards in an AGENTS.md or a CLAUDE.md, this one is worth your time. (bookmark it) Researchers measured what context compaction actually destroys across 20 product

Qwen3.8 27B is now available for both fine-tuning and Dedicated Model Inference Fine-tune it on your own data, then deploy that model on dedicated infrastructure for production, without stitching toge

This is one of the most useful writeups I have seen on keeping an LLM judge effective in production. (bookmark it) Netflix runs judges over hundreds of thousands of show-level recommendation explanati

Great paper on multi-agent systems for code review. It's challenging to know how many coding agents to use to address a problem. The default fix for weak agentic code review is more agents. In turns o

If you maintain an AGENTS.md or a CLAUDE.md, this one is worth your time. (bookmark it) Researchers traced 94K development events across 557 agentic coding sessions, plus 690K file-level change record

We’re Hiring: Member of Technical Staff (LLM Development) 🐟 https://sakana.ai/careers/member-of-technical-staff-llm-development/#English The role: ・Develop LLMs that form the foundation of Sakana AI’s

In 1948, a 32-year-old at Bell Labs published a paper nobody fully understood. Engineers found it too mathematical. Mathematicians found it too engineering-focused. One prominent mathematician reviewe

This is very nice work from NVIDIA. Like all high-performing approaches on ARC-AGI-3, it uses deep learning-guided on-the-fly synthesis of symbolic world models, i.e. navigating the world by generatin

Hardmaru and Breaking911 posted a video (14 hours ago) of a humanoid robot training for the “Robot Olympics” in Beijing. In the footage, the robot accelerated too fast, failed to brake, collided with

Banger paper on harness continual learning. (bookmark it) If you already are allowing your agents to rewrite their own prompts, skills, or memory files, this one is worth your time. (bookmark it) Cont

Moderna/Merck just ran a 1,137-patient Phase 3 trial where every single dose was unique to that patient's tumor. It worked. The pipeline: surgical resection → whole exome + RNA sequencing → ML neoanti

Top 15 models in Agent Arena vs. median task cost: who's delivering the most for their price? Key findings: Even within the top 5, median cost per task ranges from 0.62 for Kimi K3 (Max) to 3.37 for C

We will continue to offer Zero Data Retention for frontier models. As AI takes on longer, more autonomous work and delivers greater value to businesses, safety systems also need to identify risks acro

If alignment issues are becoming big enough that OpenAI is willing to commit 20% of research inference compute to chain-of-thought monitoring, that suggests that alignment issues are becoming a pretty

We have paused some frontier RL training to ensure that we can meet the appropriate alignment, security and monitoring standards for the new level of capabilities in front of us. Model progress is now

We’re sharing the concrete changes we’re making to strengthen monitoring, security, and alignment as capabilities advance. We’ve introduced stronger workload and network isolation, continuous security

Announcing AItonomy Foundation. We are PhD students and postdocs at Stanford, Princeton and Oxford, building a nonprofit research organization for AI4Science and Science4AI — and connecting early-care

A consequence of how frontier models are trained is motivated reasoning, a phenomenon well studied in humans and discussed in this podcast from @PalisadeAI. Our thoughts, beliefs and reasonings tend t

A great read if you an AI dev. Current context compactors retain 17% of the standing rules users give them. Session Constraints are instructions like 'do not delete any emails until I confirm' that bi

A 36‑page ArXiv whitepaper titled **ExtractBench** was released by Jerry Liu (jerryjliu0), describing a large‑scale, schema‑guided benchmark for real‑world document extraction from complex enterprise

Live now: our Memory & Continual Learning Track from AI Engineer World's Fair 2026. Thesis: we scaled intelligence and got the world's smartest novice. https://www.youtube.com/watch?v=iqloyWCGYQQ&list

We wrote a 36-page ArXiv whitepaper on ExtractBench 🧑‍🔬 , our effort to create the most comprehensive, schema-guided, real-world document extraction benchmark. It’s extremely detailed and covers every

Why does CLAUDE.MD keep growing? If you maintain a CLAUDE.md or an AGENTS.md, this one is worth your time. (bookmark it) This work traces why these files grow without bound. Appending an instruction i

With the smartest person I know, @HarshSensei, we're building evsys-sdk for the community, this open-source repository allows anyone to build their own continual learning system with first-class suppo

Really important point and nicely explained! Two further points worth noting: (1) if an AI system relies on a harness (which as Gary notes in other writing is increasingly improtant in top models), th

A lot of the questions we get from developers are about the concepts behind the API: TTFT, context windows, sampling, fine-tuning, quantization, deployment tradeoffs. We added Learn to the Together do

On August 7, 2026 Ethan Mollick tweeted that “every remaining good AI benchmark score has an implied asterisk next to it which reads: * could be significantly higher with a better harness.” The commen

In the era of base LLM scaling (2022-2024), I believed the LLM line of research would reach a capability plateau (as later seen with base LLMs). In late 2024, after the o3 test-time compute demo, I ch

In this episode of @wandb's Gradient Dissent , @l2k and Fireworks CEO @lqiao discuss why she believes the industry is at a turning point... ... one that calls for more open intelligence, not less. The

Just saw that the LLMs-from-scratch repository passed 100,000 stars on GitHub! This is super cool and motivating. I am really happy to see that this open-source repo has helped so many people. Thanks

this talk by openai researchers going through hugging face incident is totally insane, so much to unpack openai only realized it was their agent who hacked hugging face infra while asking hf to revoke

Taalas buried the lede for the amazing demo of their first tape out. 15k tokens per second with a llama 8b model, ability to scale that up etched onto silicon As models satisfice etching makes sense,

Also I think AISI is a great model of a government agency tasked with AI security. They have open benchmarks, very fast testing, and clear communication about incidents that is neither hyped up nor hi

🚩🚩🚩 OpenAI is 'slowing down to enhance security' after discovering swarms (!) of agents started secretly coordinating MONTHS ago 1) It started May 7 - not July 2) 'The agents discovered they could lea

Yes, the AIs were given a cybersecurity challenge, with internet access enabled and safety filters disabled. But the extent to which Mythos 5 pursued its mission (fake identities, social engineering,

If you missed today's K3 on Fireworks webinar, we've got you covered. We'll be partnering with @arizeai to chat more models on Thursday, August 6th. Sign up below! Next session, August 6 we'll be co l

arXiv:2409.03901v4 Announce Type: replace Abstract: Remote sensing (RS) image classification is central to Earth observation, but onboard deployment requires models that are accurate, efficient, and r

Picking the right agent harness is now a crucial skill for any AI engineer. Imagine using the same model, same task, and same prompt. Now move it between two agent harnesses and the cost per success c

Kimi K3 has set a new bar for OSS model intelligence! 2.8T params, 1M context, OpenAI-compatible API. Complete guide to running Kimi K3 on Together AI 👇 👏👏 @Kimi_Moonshot 👏👏 https://www.together.ai/bl