I think there are lots of good reasons to worry about whether AI is giving correct answers (especially free models), but, given the contradi…

I think there are lots of good reasons to worry about whether AI is giving correct answers (especially free models), but, given the contradiction with other recent research suggesting AI is now doing well at financial advice, I was curious about what the methodology was here. I downloaded the report, which is from a company selling different AI financial services products. Given the limited information in the report, it is impossible to judge it's accuracy, since there are only limited examples of questions. However, I am curious if anyone familiar with UK tax law knows whether GPT-6 Pro's defense of Haiku's answer is right, or if Saturn's critique is. AI chatbots give wrong answers to financial queries ‘most of the time’ https://ft.trib.al/KxRXFO6

It is one reason why I think the push for smaller, local models is more complicated than people think. If you want good answers, especially …
All benchmarks are flawed, but GPQA has been fairly consistent & highly correlated with other measured benchmars. I think it's a good way to…
I think Epoch does a great job benchmarking, but I continue to believe that open weights models are much more fragile, especially out-of-dis…
There's been a talk about how LLMs are only advancing in verifiable areas like math or coding, but that isn't what the data suggests. As mod…
This paper shows people are asking a lot of medical questions of AI already, but we have little evidence of how good or bad this is. Most of…

Source: Ethan Mollick (X) | 2026-09-19

Related