LLM-Guided Task- and Affordance-Level Exploration in Reinforcement Learning

arXiv:2509.16615v2 Announce Type: replace Abstract: Reinforcement learning (RL) is a promising approach for robotic manipulation, but it can suffer from low sample efficiency and requires extensive exploration of large state-action spaces. Recent methods leverage the commonsense knowledge and reasoning abilities of large language models (LLMs) to guide exploration toward more meaningful states. However, LLMs can produce plans that are semantically plausible yet physically infeasible, yielding unreliable behavior. We introduce LLM-TALE, a framework that uses LLMs' planning to directly steer RL exploration. LLM-TALE integrates planning at both the task level and the affordance level, improving learning efficiency by directing agents toward semantically meaningful actions. Unlike prior approaches that assume optimal LLM-generated plans or rewards, LLM-TALE corrects suboptimality online and explores multimodal affordance-level plans without human supervision. We evaluate LLM-TALE on pick-and-place tasks in standard RL benchmarks, observing improvements in both sample efficiency and success rates over strong baselines. Real-robot experiments indicate promising zero-shot sim-to-real transfer. Code and supplementary material are available at llm-tale.github.io.

Related

• Sample-Efficient Neurosymbolic Deep Reinforcement Learning
• WOMBET: World Model-based Experience Transfer for Robust and Sample-efficient Reinforcement Learning
• TimeRewarder: Learning Dense Reward from Passive Videos via Frame-wise Temporal Distance
• ACDC: Adaptive Curriculum Planning with Dynamic Contrastive Control for Goal-Conditioned Reinforcement Learning in Robotic Manipulation
• Specificity-aware reinforcement learning for fine-grained open-world classification

Source: arXiv cs.RO | 2026-04-15