Research
DeepTest Tool Competition 2026: Benchmarking an LLM-Based Automotive Assistant
arXiv:2604.12615v1 Announce Type: new Abstract: This report summarizes the results of the first edition of the Large Language Model (LLM) Testing competition, held as part of the DeepTest workshop at
arXiv:2604.12615v1 Announce Type: new Abstract: This report summarizes the results of the first edition of the Large Language Model (LLM) Testing competition, held as part of the DeepTest workshop at ICSE 2026. Four tools competed in benchmarking an LLM-based car manual information retrieval application, with the objective of identifying user inputs for which the system fails to appropriately mention warnings contained in the manual. The testing solutions were evaluated based on their effectiveness in exposing failures and the diversity of the discovered failure-revealing tests. We report on the experimental methodology, the competitors, and the results.
Related
- BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation
- When More Thinking Hurts: Overthinking in LLM Test-Time Compute Scaling
- Daily and Weekly Periodicity in Large Language Model Performance and Its Implications for Research
- Hallucination as output-boundary misclassification: a composite abstention architecture for language models
- Human-like Working Memory Interference in Large Language Models
Source: arXiv cs.AI | 2026-04-15