Beyond Black-Box Labels: Interpretable Criteria for Diagnosing SubjectiveNLP Tasks

arXiv:2604.17022v1 Announce Type: new Abstract: Subjective NLP datasets typically aggregate annotator judgments into a single gold label, making it difficult to diagnose whether disagreement reflects unclear criteria, collapsed distinctions, or legitimate plurality. We propose a schema-level diagnostic for auditing expert-designed annotation schemas prior to gold-label commitment, using only multi-annotator criterion judgments. The diagnostic separates two failure modes: unstable criteria with hard-to-operationalize boundaries, and systematic overlap that blurs the boundaries between mutually exclusive categories. Applied to persuasive value extraction in commercial documents, we find that disagreement is not diffuse: instability concentrates in a few criteria, while nearly half of covered sentences activate multiple categories. These signals align with where domain experts disagree, yielding an evidence-based audit for tightening guidelines, revising category structure, or reconsidering the annotation paradigm.

Related

• ReasonScaffold: A Scaffolded Reasoning-based Annotation Protocol for Human-AI Co-Annotation
• Annotation Entropy Predicts Per-Example Learning Dynamics in LoRA Fine-Tuning
• From Prediction to Justification: Aligning Sentiment Reasoning with Human Rationale via Reinforcement Learning
• Explain the Flag: Contextualizing Hate Speech Beyond Censorship

Source: arXiv cs.CL | 2026-04-21