Safety
Policy-Invariant Reward Shaping from LLM Feedback: A Framework for Hybrid RL Agents
arXiv:2608.18008v1 Announce Type: cross Abstract: Combining large language models with reinforcement learning is increasingly explored, yet the theoretical status of LLM-derived reward signals is ofte
arXiv:2608.18008v1 Announce Type: cross Abstract: Combining large language models with reinforcement learning is increasingly explored, yet the theoretical status of LLM-derived reward signals is often left implicit. We formalize the hybrid LLM-planner and RL-controller architecture as a Goal-Augmented Markov Decision Process and show that when the LLM per-state progress score is used as a bounded potential function, the resulting shaping term preserves the optimal policy set even when the LLM scores are inaccurate. This guarantee is stronger than what general LLM-as-reward approaches provide. We verify the result numerically on a small MDP under four potential configurations, including an adversarial one scaled to twenty times the base reward magnitude.
Related
- The Past Is Not Past: Memory-Enhanced Dynamic Reward Shaping
- In-Context Reward Adaptation for Robust Preference Modeling
- ProcessThinker: Enhancing Multi-modal Large Language Models Reasoning via Rollout-based Process Reward
- Automating Potential-based Reward Shaping with Vision Language Model Guidance
Source: arXiv cs.AI | 2026-08-19