Learning Lifted Action Models from Unsupervised Visual Traces

arXiv:2604.19043v1 Announce Type: new Abstract: Efficient construction of models capturing the preconditions and effects of actions is essential for applying AI planning in real-world domains. Extensive prior work has explored learning such models from high-level descriptions of state and/or action sequences. In this paper, we tackle a more challenging setting: learning lifted action models from sequences of state images, without action observation. We propose a deep learning framework that jointly learns state prediction, action prediction, and a lifted action model. We also introduce a mixed-integer linear program (MILP) to prevent prediction collapse and self-reinforcing errors among predictions. The MILP takes the predicted states, actions, and action model over a subset of traces and solves for logically consistent states, actions, and action model that are as close as possible to the original predictions. Pseudo-labels extracted from the MILP solution are then used to guide further training. Experiments across multiple domains show that integrating MILP-based correction helps the model escape local optima and converge toward globally consistent solutions.

Related

• WOMBET: World Model-based Experience Transfer for Robust and Sample-efficient Reinforcement Learning
• Controlling Multimodal Conversational Agents with Coverage-Enhanced Latent Actions
• Sample-Efficient Neurosymbolic Deep Reinforcement Learning
• Improving the Reasoning of Multi-Image Grounding in MLLMs via Reinforcement Learning

Source: arXiv cs.AI | 2026-04-22