Frontier models are bad at hearing ASMR: 8 judges vs human labels on 480 clips [P]

I'm training text2asmr, a text-to-ASMR model (whispering plus trigger sounds like tapping, crinkling, brushing). The generator needs every training clip tagged with the sounds in it, and there are about 126k recordings, so I tried frontier models as labellers. They were much worse than I expected, so here are the numbers. Setup. Every judge got the same prompt and the same 20 labels on 480 clips: External: sound-effect clips from FSD50K and ESC-50 that humans had already labelled with one class, 40 per class. This measures recall. People: 6-second ASMR clips labelled on our site. A class counts as present when at least half of a clip's labellers ticked it. This measures F1. Judge Trigger F1 vs people Gemini 3.1 Pro 0.325 Gemini 2.5 Pro 0.290 GPT Audio 1.5 0.288 GPT Audio 0.264 MiMo v2.6 Pro 0.243 Gemini 3.8 Flash 0.212 Claude Opus 5.5 (spectrogram image) 0.121 Grok 4.7 (spectrogram image) 0.110 Gemini 3.1 Pro cost 0.0088 a clip, MiMo 0.0008. What surprised me: Clean sound effects are not ASMR. On the FSD50K/ESC-50 brushing clips Gemini 3.1 Pro named brushing 92% of the time. On real ASMR clips its brushing F1 against people was 0.18. Paper is nearly invisible. On 40 paper and page-turning clips, Gemini 3.1 Pro and 2.5 Pro each named paper rustling 4 times. MiMo named it once. Voices are fine. Whispering F1 was 0.70 to 0.79 for the Gemini and MiMo judges. The physical triggers are where it falls apart. Each judge has a favourite wrong answer. On the external clips Gemini 2.5 Pro said "sticky" 59 times where nothing sticky was labelled. Check that labels are learnable before training on them. An earlier round used a cheap model that also saw the transcript. A linear probe on frozen CLAP embeddings couldn't beat the majority class on its labels (51% on tapping vs page turning). It was reading the transcript, not listening. What I do with it. All the judges, plus an AudioSet AST tagger, an energy detector and our own CLAP model, go into Dawid-Skene, with each judge weighted per class by its measured reliability against people. Clips where the fused probability is unsure go to people first, and the classes no judge hears (paper, brushing, cutting, sticky, spraying) are served before anything else. The student is CLAP-ASMR, a fine-tune of laion/clap-htsat-unfused. Adding 2,400 Freesound clips took test mAP from 0.516 to 0.549, but it still detects brushing and paper on 0% of the external clips at p > 0.5, because the fusion step can't keep labels that no judge can hear. That's the gap people have to fill. The ask. If you have headphones and five minutes: https://asmrboard.vercel.app. You get 6-second clips and tick what you hear. Each clip goes to several people independently, and known clips and repeats are mixed in to measure reliability. You need a free account for now (email and password), and you have to be 18+ because some clips are intimate vocal sounds. So far 34 people have given 1,056 labels. The weights and the labels will be released when the model is trained. Code: https://github.com/aloshdenny/text2asmr Has anyone had better luck getting audio LLMs to hear fine-grained foley? I'd also take suggestions for sound-event models worth adding as judges. submitted by /u/MissAppleby [link] [comments]

Source: r/MachineLearning | 2026-10-22