GIST: Generator-state-Informed Supervision for Tabular Foundation Models

arXiv:2610.12904v1 Announce Type: new Abstract: Tabular foundation models (TFMs) based on prior-data fitted networks typically use sampled hard query labels, leaving avoidable sampling variability in pretraining. A key observation is that intermediate generator states can reveal label uncertainty beyond a sampled outcome. We propose Generator-state-Informed Supervision for TFMs (GIST), which uses these states to average query losses over conditional label distributions. GIST jointly designs task generation and query supervision, using entropy-calibrated encodings to obtain tractable, input-consistent targets on supported label-dependent branches. We show that this averaging preserves the population objective and yields no larger gradient covariance at fixed parameters within each generating distribution. Under matched pretraining schedules, GIST improves mean rank across three architectures on both TabArena and TALENT without changing architecture or inference. Across these benchmarks and architectures, GIST wins 62.1% of paired dataset-level comparisons against the corresponding hard-label controls.

Related

• Tydra: An Efficient Hybrid Model for Tabular Data
• VIP-COP: Context Optimization for Tabular Foundation Models
• TabPFN-MT: A Natively Multitask In-Context Learner for Tabular Data
• Understanding Latent Flow Models for Tabular Data Synthesis: Targets, Paths, and Sampling
• Towards Evaluating Data Priors for Tabular Foundation Models

Source: arXiv cs.LG | 2026-10-12