TAFFY: A Task-Adaptive Tabular Foundation Model with In-Context Diversity

arXiv:2610.07559v1 Announce Type: new Abstract: Recent progress in tabular foundation models suggests that training on synthetic tasks can substantially improve in-context learning capabilities, with overall performance largely depending on how well models can infer task-specific predictive relationships from the available context during inference. In this paper, we introduce TAFFY, a tabular foundation model with an In-Context Diversity Prior and a Task-Conditioned Looped Transformer that strengthen this ability. Specifically, to construct each synthetic pretraining context, the In-Context Diversity Prior samples from multiple related environments derived via controlled interventions and distribution shifts on a shared causal process. This in-context diversity encourages the model to learn a more comprehensive and task-specific representation. Moreover, the Task-Conditioned Looped Transformer iteratively and selectively applies a shared group of Transformer blocks to refine contextual representations, with a task-conditioned gate modulating the final hidden-state update. This enables task-adaptive iterative refinement. Together, these components encourage the model to identify predictive relationships from contextual contrasts during pretraining and dynamically modulate context integration for each task. Across six classification and five regression benchmark datasets, TAFFY attains the lowest average rank.

Related

• From Synthetic Priors to Model Behavior: Structural Coverage in Tabular Foundation Models
• TabDPT-Turbo: Efficient In-Context Learning for Tabular Prediction
• Closing the Context Gap: Activation Alignment for Tabular In-Context Learning
• Architecture Alignment With Sparse Priors in Tabular Foundation Models
• Align-RAG: Alignment Is All You Need for TSFM In-Context Learning

Source: arXiv cs.LG | 2026-10-07