Improving Few-Step Language Flows with Untied Self-Conditioning

arXiv:2608.22244v1 Announce Type: cross Abstract: Flow-matching language models refine all token positions in parallel and can trade sampling steps for latency, yet generation quality still degrades sharply with few sampling steps. We trace a source of this degradation to a train--inference mismatch in previous-prediction self-conditioning: during training, the self-conditioning input is computed from the current noisy state with no intervening solver step; during sampling, the solver folds the previous prediction into the latent before that same prediction reappears as the explicit self-conditioning input. This coupling, absent during training, creates redundancy that grows with step width. We show that the mismatch degrades both the self-conditioning input and the solver update, and derive a correction for each from the model's own structure. From the frozen projection weights we identify directions along which the self-conditioning input is redundant with the latent and dampen them; from the solver's integration structure we derive that a step-average prediction is needed and approximate it from prediction history, with scale set by offline trajectory statistics. The resulting sampler, Untied Self-Conditioning, requires no retraining and uses one evaluation per step. At 8 sampling steps on LangFlow, it reduces OpenWebText generative perplexity from 531 to62 (8.6imes); under an adapted Arena-Hard-Autov2 protocol, its outputs are preferred in 96% of pairwise comparisons. On ELF-B it reduces generative perplexity from 71 to~43. Improvements hold from 8 to 256 sampling steps.

Source: arXiv cs.AI | 2026-08-25