the deeper point: model choice and harness choice aren't independent variables. benchmarking a model without specifying the harness is like …

the deeper point: model choice and harness choice aren't independent variables. benchmarking a model without specifying the harness is like benchmarking a chip without specifying the compiler. the number means something, but not what people think it means. profiles make the harness a first-class, named object — you can diff it, version it, and swap it when you swap models. we're continuing to see clear examples where a model's harness is a major determinant of overall performance. with the same model, running on same task, it's easy to observe very different scores depending on (system) prompts, tools (& their descriptions), and middleware (steerin…

Source: Harrison Chase (X) | 2026-05-06