Towards a Linguistic Evaluation of Narratives: A Quantitative Stylistic Framework

arXiv:2604.19261v1 Announce Type: new Abstract: The evaluation of narrative quality remains a complex challenge, as it involves subjective factors such as plot, character development, and emotional impact. This work proposes a quantitative approach to narrative assessment by focusing on the linguistic dimension as a primary indicator of quality. The paper presents a methodology for the automatic evaluation of narrative based on the extraction of a comprehensive set of 33 quantitative linguistic features categorized into lexical, syntactic, and semantic groups. To test the model, an experiment was conducted on a specialized corpus of 23 books, including canonical masterpieces and self-published works. Through a similarity matrix, the system successfully clustered the narratives, distinguishing almost perfectly between professionally edited and self-published texts. Furthermore, the methodology was validated against a human-annotated dataset; it significantly outperforms traditional story-level evaluation metrics, demonstrating the effectiveness of quantitative linguistic features in assessing narrative quality.

Related

• ReasonScaffold: A Scaffolded Reasoning-based Annotation Protocol for Human-AI Co-Annotation
• LQM: Linguistically Motivated Multidimensional Quality Metrics for Machine Translation
• Calibrating Model-Based Evaluation Metrics for Summarization
• Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations

Source: arXiv cs.CL | 2026-04-22