Grounding Spatial Relations in a Compact World Model: Instruction Leakage and a Goal-Free Dynamics Fix

arXiv:2607.06925v1 Announce Type: new Abstract: Compact world models that condition on a language goal promise to ground relations such as ``put the red block left of the blue block'' using a sparse set of explicit reference anchors. We ask when such references actually ground a relation, and identify a trap: a goal-conditioned predictor reaches a striking 0.90 relation-readout accuracy, yet this is instruction transcription, not perception. Withholding the goal collapses it to chance (0.90!o!0.27, three seeds) and a counterfactual instruction makes the predicted anchors follow the false instruction 94.5% of the time (true scene 2.3%; N{=}256). Tested across three settings and a within-task ablation, our central claim characterizes the confound: extbf{instruction leakage occurs when the scored quantity is transcribable from the instruction (when the instruction names the answer) and is essentially independent of how predictive the non-instruction inputs are.} Our tabletop and the external BabyAI benchmark leak, whereas a Language-Table forward-dynamics world model whose instruction names referents does not, until the instruction is augmented to name the direction; and degrading the action never increases leakage, the opposite of what predictor-competition predicts. The diagnosis prescribes the fix: keep the goal out of the dynamics (it belongs to the planner's cost) and supervise the read path, recovering genuine, instruction-independent grounding (0.88, identical with and without the goal). The detection protocol and remedy apply to any goal-conditioned world model whose instruction names the scored quantity.

Source: arXiv cs.AI | 2026-07-09