Challenges of Auditing: Variability in Outputs of Large Language Models for Health

arXiv:2609.16590v1 Announce Type: new Abstract: People increasingly use frontier AI models for health advice, but via different access modes (e.g., ChatGPT, ChatGPT Health, APIs) with varying settings. Here, we find systematic differences across access modes. Because evaluations typically rely on APIs while consumers interact through chatbot interfaces, these discrepancies limit evaluation validity. Our findings underscore an urgent need for model providers to enable faithful replication of consumer experiences and settings for rigorous audits.

Related

• Language Shapes Mental Health Evaluations in Large Language Models
• Self-Consistency from Only Two Samples: CoT-PoT Ensembling for Efficient LLM Reasoning
• Attention Sinks as Internal Signals for Hallucination Detection in Large Language Models
• Not What, But How: A Communicative Audit of LLM Response Framing

Source: arXiv cs.CL | 2026-09-16