On-the-fly Weight Generation: A Hypernetwork Proof of Concept on ARC-1D

arXiv:2610.00820v1 Announce Type: cross Abstract: General-purpose models can adapt to many tasks from context, while specialised models can execute individual functions with less capacity. Yet obtaining such specialists requires task-specific training or adaptation. We ask whether they can instead be generated directly from a few demonstrations. Using ARC-1D as a controlled testbed, we show that individual transformations can be represented by tiny specialist models, and that a hypernetwork can generate their parameters from context. The generated parameters form a structured weight space, while the resulting specialists show partial compositional generalisation and generalisation to transformations not seen during training. In both settings, removing explicit task identifiers improves generalisation beyond the training transformations. Together, these results provide a proof of concept that few-shot task context can be compiled on-the-fly into compact executable model parameters, and that the resulting weight space can support reuse and generalisation beyond known functions.

Related

• ThinkSwitch: Context Distillation with LoRA and Weight Interpolation for Specific-Purpose Reasoning Tasks
• MMAI Gym for Science: Training Liquid Foundation Models for Drug Discovery
• How Do Language Models Compose Functions?
• Dynamic Context Adapters: Efficiently Infusing History into Vision-and-Language Models

Source: arXiv cs.AI | 2026-10-02