Prinz has been running legal benchmarks against AI systems and the latest models are very good... but cheaper/low reasoning models are not v…

Prinz has been running legal benchmarks against AI systems and the latest models are very good... but cheaper/low reasoning models are not very good. So, how do you make sure your vendor selling an AI solution is incentivized to serve the good (and therefore expensive) models? Reading a memo produced by the client's legal department with extensive help from AI: - looks very pretty; beautiful font; nice formatting; there's even a handsome-looking (although unnecessary) organizational chart included - completely incorrect conclusion on whether a regulato…

Thresholds (like “can a robot make a good coffee?”), are not actually very good benchmarks because you can’t track progress towards a goal.
As much as the state of benchmarks in AI is flawed, it is so much easier to track AI progress than robotics. Not sure what you can make of a…
There are more competitive small model makers, but there is still a very big gap between what small models can do and what large models can …
'Switch to a cheaper model to save money' is a problem because cheaper models are worse (maybe they are good enough for a particular purpose…

Source: Ethan Mollick (X) | 2026-09-02

Related