the independence of AI evaluators is going to matter more... so i made it a 'benchmark': http://evaluatorbench.com (research preview) source…

the independence of AI evaluators is going to matter more... so i made it a "benchmark": http://evaluatorbench.com (research preview) source-linked directory of third-party evaluators of frontier AI. each one has an independence score you can take apart: change the weights, change which evidence counts (standard, against-interest, primary-only). not quality or competence. independence only and then i compared where we are to other highly regulated industries. one thing that stood out is that in most regulated industries, the regulators monitor the assessors too. if the pattern holds, regulators would oversee METR et al., not only the labs still a preview, not complete, not citable yet. i burned a stupid amount of agent tokens on it. contributions, remixes, forks, benchmaxxing all welcome 😉

Source: Yohei Nakajima (X) | 2026-09-16