MIKASA-Robo-VLA: Benchmarking Memory in VLA Models for Long-Horizon Manipulation

arXiv:2610.00604v1 Announce Type: cross Abstract: Vision-language-action policies often see only one or a few recent frames, which makes it difficult to evaluate how they use information that disappears during a task. We introduce MIKASA-Robo-VLA, a benchmark of 90 language-conditioned manipulation tasks. All but 10 hide the cue an action depends on. Those 10 are reactive controls. MIKASA-Robo, the suite it rebuilds, has 32 tasks and uses language only in a representative VLA subset. Here every task provides an instruction, while memory-dependent tasks hide a task-relevant cue and reactive controls keep it available. For 70 tasks, environment phase timings specify an information gap, and for 28 of them the gap exceeds the 16-frame window of the widest fixed-context VLA we survey. The gap counts only the interval the cue is provably absent, not the full duration a policy must retain it, so every memory-dependent task still requires memory by construction, including the ones whose measured gap is short. We release 22,500 oracle trajectories across 10 memory types in RLDS and LeRobotDataset v3. A reference pi_{0.5} baseline with current images and proprioception, but no observation history or explicit memory module, is fine-tuned on 14 tasks and achieves 0.211 pm 0.044 mean task success. Its lower success on the evaluated Long-split tasks is confounded by open-loop chunking and the memory types represented in that subset. Project page: https://mikasarobo.github.io/

Related

• Divide-and-Remember: Recursive Action-Relevant Memory for Long-Horizon VLA Policies
• HELM: Harness-Enhanced Long-horizon Memory for Vision-Language-Action Manipulation
• RoboMME: Benchmarking and Understanding Memory for Robotic Generalist Policies
• Benchmarking Vision-Language-Action Models on SO-101: Failure and Recovery Analysis

Source: arXiv cs.AI | 2026-10-02