KnowsTFM: Knowledge-Informed Fine-Tuning of Small Tabular Foundation Models

arXiv:2606.30258v1 Announce Type: cross Abstract: Tabular foundation models have advanced deep learning for tabular data by delivering strong default performance across many small and medium tasks. Yet in niche domains, where data is scarce, high-dimensional, and shifted from the pretraining distribution, they may still fail to outperform carefully designed domain-specific methods. Many such domains also provide curated relational knowledge in the form of knowledge graphs and knowledge banks, but how to use this knowledge to improve and steer extit{small} specialist tabular foundation models remains unclear. We address this problem through extbf{Know}ledge-informed fine-tuning of extbf{s}mall extbf{T}abular extbf{F}oundation extbf{M}odels (modelname). Specifically, we study nanoscale TabPFN- and TabICL-style variants, pretrained under controlled synthetic prior families and adapted using two complementary mechanisms: structural attention priors derived from knowledge graphs and parameter-efficient low-rank updates. We show that injecting domain-specific structural knowledge during fine-tuning yields meaningful gains over vanilla variants in specialist settings, whereas gains on general-domain tasks are marginal. We further observe that continual fine-tuning of frontier models can trigger collapse of pretrained knowledge and mechanisms.

Source: arXiv cs.AI | 2026-06-30