Safety
ASSERT: A Measurement Pipeline for GenAI Audits
arXiv:2608.13840v1 Announce Type: cross Abstract: Audits of generative AI (GenAI) systems often summarize behavior as a reported rate: how often the audited system complies with policy. Researchers an
arXiv:2608.13840v1 Announce Type: cross Abstract: Audits of generative AI (GenAI) systems often summarize behavior as a reported rate: how often the audited system complies with policy. Researchers and stakeholders use that rate to compare systems, track regressions, and gate deployment. A reported rate reflects both the system under audit and the measurement choices behind it, so a change in the rate can leave it unclear whether the system or those choices moved. We introduce ASSERT, a specification-driven measurement pipeline for GenAI audits that ties each reported rate to a written specification of the measurement choices used to produce it. ASSERT helps draft a behavioral rubric and test cases, then runs the audit against a GenAI system and returns a reported rate. In a case study on conversational deception, we observe that the reported rate moves substantially with the dialogue setup, the simulated user, the judge, and the evidence bar for non-compliance. These measurement choices substantially change the reported rate and can reorder GenAI system rankings. Because each reported rate is tied to an explicit specification, differences across audits are easier to attribute and interpret.
Related
- Ethics Testing: Proactive Identification of Generative AI System Harms
- AI-Assisted Systematization for Evaluating GenAI Systems
- A Chain Is Only as Strong as Its Weakest Link: A Scoping Review of System Integration Audits in AI
- Fairness Is Not Enough: Auditing Competence and Intersectional Bias in AI-powered Resume Screening
Source: arXiv cs.AI | 2026-08-17