Does Learning Recurrent Networks Matter in Partially Observable RL?

arXiv:2610.13197v1 Announce Type: cross Abstract: Reinforcement learning (RL) under partial observability typically re- lies on recurrent networks to encode temporal information, with the recurrent module trained jointly with the policy or value function. This raises a natural question: is learning the recurrent network actually necessary? In this paper, we investigate whether a frozen recurrent network can provide sufficient memory for solving par- tially observable Markov decision processes (POMDPs). Motivated by the theoretical foundations of Reservoir Computing (RC) and Generalized Reservoir Computing (GRC), we study a simple archi- tecture that combines a fixed recurrent network with a trainable nonlinear output module. We evaluate this approach across tasks with varying memory demands, observation modalities, and action spaces. Surprisingly, frozen recurrent networks support recall over hundreds of steps and achieve competitive control performance across diverse POMDPs, while using less than half the number of trainable parameters required by their fully trainable counterparts. These results suggest that learning the recurrent memory module is not always necessary for solving POMDPs. Frozen recurrent networks therefore provide a simple, parameter-efficient, and sur- prisingly strong baseline for partially observable RL. Our source code is available at https://github.com/Aphlios/FRN-POMDP

Related

• Adversarial Latent-State Training for Robust Policies in Partially Observable Domains
• LLM Agents for Deliberative Collaboration: A Study on Joint Decision Making Under Partial Observability
• Lindblad-Inspired Multi-Timescale Reservoir Computing with Separable Rotation and Dissipation

Source: arXiv cs.AI | 2026-10-12