A Computational Model of the Hippocampus Supports Exploratory Behaviour in Reinforcement Learning Agents

Abstract

The hippocampus supports spatial navigation, memory, and planning through the formation of a cognitive map: a structured environmental representation reflected in its neural activity. These neural dynamics can be modeled computationally using recurrent neural networks (RNNs). However, these RNNs are typically trained via reinforcement learning (RL) using external rewards, failing to capture the intrinsic drive of freely exploring organisms in the absence of external rewards. Instead, reward-free RL models, which rely on internal environmental representations, are better candidates to study novelty-seeking and exploratory behaviour. This study aims to investigate whether an RNN exhibiting hippocampal-like activity builds spatial representations sufficient to support exploratory behavior in reward-free RL agents. We leveraged an existing RNN trained for sensory sequence prediction, which exhibits hippocampal-like activity patterns, and used its prediction error as the intrinsic reward to train an Actor-Critic agent. We used a Novel Object Recognition task to quantify its preference for novel stimuli. The RL agent occupied the region of interest (defined as a 3-unit radius around the novel object) significantly more often than a random agent (29.3% ± 3.4% vs. 11.3% ± 1.1% [mean ± SEM], Welch's t-test, p < 0.01), across multiple episodes and novel object locations. The RL agent’s performance was also measured in a multi-room environment. It spent 8.2% ± 0.2% of its time in the most distant and novel room, a significant increase over the 0.023% ± 0.017% of the random control (p < 0.05). This work demonstrates that hippocampal-like representations can support exploratory behavior, and can help us investigate how cognitive maps guide exploration and navigation.