Applications
Social science has spent the last century figuring out solutions to the problems facing AI benchmarking today, you don’t need to reinvent ev…
Social science has long addressed the challenge of evaluating qualities that cannot be objectively verified—such as writing quality or creative ideas—and its methodologies remain relevant for today’s
Social science has long addressed the challenge of evaluating qualities that cannot be objectively verified—such as writing quality or creative ideas—and its methodologies remain relevant for today’s AI benchmarking problems. For non‑verifiable domains, human opinions serve as a primary benchmark: researchers gather and analyze qualitative data to determine if an output meets real‑world standards. Therefore, AI practitioners should adopt established social‑science approaches, notably qualitative research methodology, to assess systems where objective metrics are insufficient.
Related
- The inability of AI models to produce creative variation is a huge gap. The fact that they generate similar ideas limits their ability to do…
- The AI replies really suck the whimsy from weird posts.
- Math is easy* because it has verifiable outputs and few messy judgement choices to make. Which AI labs have the guts to make advancing socia…
- As the benchmarks that test frontier AI on get more complex, we are losing one of the most important aspects of benchmarking: comparisons to…
Source: Ethan Mollick (X) | 2026-08-16