Search: “benchmark”

resharing this note, find it helpful given all the great open evals work + teams building vertical agents Evals are a proxy for the behavior we want our agent to exhibit in production Model+Harness pu

Today we’re releasing Qwen-Scope 🔭, an open suite of sparse autoencoders for the Qwen model family. It turns SAE features into practical tools： 🎯 Inference — Steer model outputs by directly manipulati

I built a TPU-native medical Q&A fine-tuning pipeline using Gemma 3 with Keras and JAX The project fine-tunes Gemma-3 on medical dialogue data from ChatDoctor and evaluates it on MedMCQA, with a large

We're open-sourcing Hy-MT1.5-1.8B-1.25bit — a 440MB translation model that runs fully offline on your phone, supports 33 languages, and outperforms Google Translate. At 1.8B parameters, it matches com

.@huggingface unveiled ml-intern – an open-source agent that automates the gritty post-training loop: - reading papers - tracing citations - curating datasets - running experiments - and iterating lik

I've been trying to make transformers more agent-friendly: agentic CLI, a skill, doc rewrites, canonical examples. It felt a bit like shooting in the dark: hard to measure progress, and hard to ensure

This post criticizes Microsoft's approach to integrating AI assistants into Outlook, expressing frustration that the AI chatbot has limited functionality and capabilities despite Microsoft's ownership

Open Models & a potential looming Haiku-apocalypse? 🚀 chart comparison UI courtesy of @OpenRouter there's a large class of problems where you don't need frontier intelligence. you want cheap, fast, an

This is an incredibly cool experiment It is also fascinating that the model knows information up to 1931, but, at least in some science topics, seems very stuck in the early 1900s. For example, it def

Sebastian Raschka has updated his LLM architecture gallery with higher resolution figures and improved summaries, including coverage of the DeepSeek V4 model architecture. This resource provides visua

Great paper on improving proactive agents. (bookmark it) Proactive agents act before you do. But how do you evaluate something that's supposed to anticipate needs you haven't expressed? This work intr

Our GB300 cluster went down yesterday, just as Deepseek released 😱 We were 😥 but @CoreWeave came through to contribute to the Open Source. They scrambled in the compute crisis, finding 2 spare dev rac

Spot on 🎯 We don't need frontier intelligence to automate searches and sending emails - We don't need trillion parameter models to be able to summarize articles or technical documents - We don't need

We've got all the models here: https://dell.huggingface.co/authenticated/models Kimi K2.5, Mistral, Cohere, Arcee AI Trinity Large, Google Gemma, Meta/Llama, Qwen, Nvidia Nemotron, Grok, GPT OSS, Deep

What if instead of building one giant AI, we evolved a coordinator to orchestrate a diverse team of specialized AIs? 🐟 Excited to share our new paper: “TRINITY: An Evolved LLM Coordinator”, published

Fireworks AI posted a question to their audience asking what features or capabilities they would like to see developed next, with a specific mention of full parameter tuning as a potential option. Thi

As a point of comparison, our commercial document OCR solution LlamaParse wins on all relevant dimensions (except content faithfulness against Opus) and is 1.25c / page prior to an enterprise plan. Co

Really impressed by how smooth switching most of my coding tasks to Codex (GPT-5.5) from Claude Code (Opus 4.7) has been. I thought it was going to be more difficult and that I would be 'fighting' wit

Tesla FSD is the first AI saving lives at scale....on real roads every single day Road accidents kill 1.19 million people every year - the #1 cause of death for ages 5–29 94%+ of crashes are caused by

btw in talking to friends the best framing for how to discuss GPT-Image-2-Thinking taking multiple tens of mins for generation and being able to oneshot QR codes and diagrams and logos and foods and f

Evals ~= Environments…they’re one of the best investments a team can make for improving agents Step 0: Turn On Tracing for Agents Step 1: Point compute at Traces to understand agent behavior, segment

Ethan Mollick posted on X about having early access to GPT-5.5, commenting positively on its capabilities and noting that the Pro version is particularly strong. He indicated that a full detailed writ

Hugging Face Releases ml-intern: An Open-Source AI Agent that Automates the LLM Post-Training Workflow [The 'AI Intern' that actually ships SOTA models ] This isn't just another ML Research Loop wrapp

Kimi K2.6, a model developed by Moonshot AI, has achieved the top ranking on OpenRouter's programming leaderboard, indicating superior performance in code generation and programming-related tasks comp

Learn more about this model release https://x.com/Alibaba_Qwen/status/2046939764428009914?s=20 🚀 Meet Qwen3.6-27B, our latest dense, open-source model, packing flagship-level coding power! Yes, 27B, a

🚀 Meet Qwen3.6-27B, our latest dense, open-source model, packing flagship-level coding power! Yes, 27B, and Qwen3.6-27B punches way above its weight. 👇 What's new: 🧠 Outstanding agentic coding — surpa

Qwen 3.6 27B model is available on Ollama! Use it with all the integrations in Ollama or chat with the model. Chat with the model: ollama run qwen3.6:27b OpenClaw: ollama launch openclaw --model qwen3

arXiv:2602.11675v3 Announce Type: replace Abstract: Large language models may answer causal questions correctly for the wrong reasons, substituting associational shortcuts P(Y|X) for the interventiona

Ethan Mollick comments on an impressive AI model achievement, noting that the performance gap between models appears consistent with historical trends based on his assessment.

Introducing ml-intern, the agent that just automated the post-training team @huggingface It's an open-source implementation of the real research loop that our ML researchers do every day. You give it

Kimi 2.6 Thinking seems very good for an open weights model, but many rough edges compared to closed SoTA. The Lem Test resulted in a 74 page thinking trace... and an okay-ish answer. It did an okay T

LLMs are still not consistent judges of qualitative work, and small changes to how that work is presented affect outcomes. Better harnessing and methods (multiple judging runs with randomized orders,

The very first quadcopter prototype was the Breguet-Richet Gyroplane No. 1, created in 1907 by Breguet Aviation in France. The first consumer quadcopter drone was the Parrot AR.Drone, released in 2010

Excited to go public with Logos, a first principles system designed for intuition & innovation We are starting by sharing results in physics before moving to other domains Today’s is a lot of fun, one

Make your next Replit purchase more rewarding with Amex Offers®. Eligible American Express® Card Members can enroll and earn a 50 statement credit after they enroll and spend a minimum of 100 in one o

NEW paper from NVIDIA. EDA tools like ABC have been hand-tuned by humans for decades. New research from NVIDIA shows they can evolve themselves. The work introduces the first self-evolving logic synth

releasing qwen3.6 35b a3b carnice edition (inspired by @kaiostephens) to enjoy an hermes agent tailored finetuned on unified memory hardware! safetensors and gguf versions (q8, 6,5 and 4) available! f

This post draws a humorous parallel between using images of salarymen eating ramen as a test case for AI image generation models in Eastern contexts and the Western 'Will Smith eating spaghetti' meme,

We only have spotty information about this very important topic. It suggests AI can be good at diagnosis, but the real world doesn't always match the experiments. https://x.com/emollick/status/1980474

Anyhow, I realize nobody cares and all the AI labs have started presenting their GDPval-AA score, but it is an incredibly gameable output with low face validity and we really need trustworthy measures

Ethan Mollick observes that an AI system (likely Claude or another large language model) still declines to write sestinas, suggesting that certain behavioral constraints or limitations persist despite

Coding agents learn from experience, but that knowledge stays locked in silos. Solve a thousand SWE tasks, and none of that wisdom helps with competitive coding. What if memories could transfer across

Great to be at HumanX with @Silverside_AI @LumaLabsAI and @krea_ai The SF frontier AI builder community keeps getting stronger. We're not done yet. Keep track of when and where you can meet up with th

Today we’re announcing Ternary Bonsai: Top intelligence at 1.58 bits Using ternary weights {-1, 0, +1}, we built a family of models that are 9x smaller than their 16-bit counterparts while outperformi

VLM Performance：Qwen3.6 is natively multimodal, and Qwen3.6-35B-A3B showcases perception and multimodal reasoning capabilities that far exceed what its size would suggest, with only around 3 billion a

Banger paper from NVIDIA. Agentic reasoning needs models that are not just capable, but efficient at long-context inference. The agent model layer is moving toward open, long-context, high-throughput