All against the machine: the Solo score for rating skill in variable environments

arXiv:2610.04523v1 Announce Type: cross Abstract: We propose a distribution-free metric to rate individual skill in player-versus-environment'' settings, where participants face heterogeneous tasks without direct opponents. Such settings are common in digital platforms, games, education, finance, and the benchmark evaluation of AI agents. They combine high randomness, tasks of widely varying difficulty, and unknown heterogeneity across individuals. Our metric maps each task outcome to a bounded performance score with zero population mean and variance bounded by 1/3, regardless of the outcome distribution, so that scores are directly comparable across tasks. Aggregating these scores over tasks yields an interpretable skill score for each individual (the Solo'' rating), and a reshuffling null model tests whether that score exceeds what chance alone would produce. We validate the approach on the progression of 2imes10^5 players in the mobile games Candy Crush Saga and Bubble Witch 3 Saga. The metric identifies high- and low-skill players with high statistical confidence, their classification persists over hundreds of subsequent levels, and a windowed version of the score tracks changes in performance along progression.

Related

• Skill Coverage: A Test Adequacy Metric for Agent Skills
• Foresight Arena: An On-Chain Benchmark for Evaluating AI Forecasting Agents

Source: arXiv cs.LG | 2026-10-06