'One thing that we've been seeing recently is that inference benchmarks don't really match production workloads that well.' - @realDanFu, VP…

"One thing that we've been seeing recently is that inference benchmarks don't really match production workloads that well." - @realDanFu, VP of Kernels When you're running dozens of concurrent coding agents — each with 45k–200k token contexts — the benchmarks that matter are the ones that stress KV cache, scheduler limits, and throughput under real load. We ran those benchmarks. Our Inference Engine delivered: → 31% higher TPS than the next fastest OSS engine → 2× better time-to-first-token at saturation → 76% lower cost per request vs. Claude Opus 4.6 Read the full technical breakdown → https://togetherai.link/O0VBJR0 Media

Source: Together AI (X) | 2026-05-19