Data-efficient Targeted Token-level Preference Optimization for LLM-based Text-to-Speech

arXiv:2510.05799v2 Announce Type: replace-cross Abstract: Aligning text-to-speech (TTS) system outputs with human feedback through preference optimization has been shown to effectively improve the robustness and naturalness of language model-based TTS models. Current approaches primarily require paired desirable and undesirable samples at the utterance level. However, such pairs are often limited in TTS output data, and utterance-level formulation prevents fine-grained token-level optimization needed for accurate pronunciation alignment. In this study, we propose TKTO that eliminates the need for paired data, enabling a more data-efficient training paradigm, and directly targets token-level units, automatically providing fine-grained alignment signals without token-level annotations. TKTO improves the challenging Japanese TTS accuracy by 39% and reduces CER by 54%, automatically assigning 12.8 times stronger reward to targeted tokens.

What's In My Human Feedback? Learning Interpretable Descriptions of Preference Data
DVPO: Distributional Value Modeling-based Policy Optimization for LLM Post-Training
Influencing Humans to Conform to Preference Models for RLHF
Continuous-Utility Direct Preference Optimization
Meta-Aligner: Bidirectional Preference-Policy Optimization for Multi-Objective LLMs Alignment

Source: arXiv cs.AI | 2026-04-28

Related