Aligning Language Models for Lyric-to-Melody Generation with Rule-Based Musical Constraints

arXiv:2604.18489v1 Announce Type: cross Abstract: Large Language Models (LLMs) show promise in lyric-to-melody generation, but models trained with Supervised Fine-Tuning (SFT) often produce musically implausible melodies with issues like poor rhythm and unsuitable vocal ranges, a phenomenon we term "constraint violation". To address this, we propose a novel alignment framework that instills musical knowledge without human annotation. We define rule-based musical constraints to automatically generate a preference dataset from an SFT model's outputs. The model is then aligned through a sequential process, first using Direct Preference Optimization (DPO) on paired preference data, followed by Kahneman-Tversky Optimization (KTO) on unpaired negative samples. Experimental results demonstrate that our aligned model substantially reduces rule violations and outperforms strong baselines in both objective and subjective evaluations, generating melodies with substantially improved musicality and coherence. An interactive demo with audio comparisons is available at https://arain233.github.io/AligningMelody-demo.

Related

• The Art of (Mis)alignment: How Fine-Tuning Methods Effectively Misalign and Realign LLMs in Post-Training
• Compiling Activation Steering into Weights via Null-Space Constraints for Stealthy Backdoors
• Alignment Data Map for Efficient Preference Data Selection and Diagnosis
• What Drives Representation Steering? A Mechanistic Case Study on Steering Refusal
• Reverse Constitutional AI: A Framework for Controllable Toxic Data Generation via Probability-Clamped RLAIF

Source: arXiv cs.CL | 2026-04-21