One might assume that our memories accurately record the details of our experiences, and that remembering is like retrieving a file from something like a ‘filing cabinet’ in the brain. But in reality, memories are active reconstructions of what happened, which show classic patterns of distortion [1], are influenced by our beliefs about the world [2-3], and undergo many changes after their encoding (through a process known as consolidation) [4-5].

Brains need to make predictions to survive, e.g. to predict that food can be found in a certain location, or to predict the presence of a predator from a distant sound. Learning complex correlations between different stimuli allows predictions to be made, and to do so biological intelligence needs a way to extract statistical structure from experience. Generative neural networks, such as those used in modern machine learning, provide a mechanism for learning this structure by learning to predict their own inputs, minimising ‘prediction error’ (i.e. how ‘surprising’ their inputs are).

In our paper we bring together these ideas to explore how the brain uses memories to develop generative models of experience through a process of consolidation, and in turn how these generative models help to reconstruct experiences during memory ‘retrieval’. After episodic memories (i.e. memories of our own experiences) are initially encoded in the hippocampus, they can be replayed in fast-forward during rest and sleep [6-8], and we simulate how this process helps a larger neocortical network learn to make predictions. We then simulate how this large generative network supports episodic memory (for experiences), semantic memory (for facts), imagination, and inference.

The effects of consolidation

 Consolidation doesn’t just change which brain regions are involved in episodic memories, but also their properties.

 Memory distortions are a key source of evidence explored in the paper. Human memory is notoriously inaccurate, but errors are not random– they show patterns which provide clues to how the brain encodes and processes memories. Specifically, memories are distorted to better match our expectations about the world; for example, stories are remembered as more congruent with our beliefs [1], and scenes are remembered as distorted towards a typical viewpoint [9-10].

 Memories become more useful (e.g. for making new inferences) through consolidation [11-13] even as they show greater distortion. Thinking of memory consolidation as integration into a generative model addresses this paradox, as the assimilation into a mental model allows memories to be used more flexibly, but comes at the cost of unique details.

 Memory encoding and prediction error

 Memories bind together elements along a spectrum from coarse-grained and conceptual to fine-grained and sensory. For example, the memory of a day at the beach is likely to bind together concepts like ‘beach’ and ‘sea’ along with sensory details like the melody of an unfamiliar song, or sight of a particular sandcastle. Evidence for this includes the existence of ‘concept cells’ in the hippocampus, neurons which fire in response to abstract concepts [14]. (In other words, a concept cell might respond to the name, face, and voice of a particular person, despite their very different sensory content.) Relying partly on concepts is a more compact way to store memories than recording every sensory detail – the concept of ‘beach’ predicts many aspects of the scene, allowing the brain to reconstruct the appearance of the day out more efficiently than by storing every ‘pixel’ of sea and sand.

 We explore how the hippocampus could encode memories as a combination of ‘conceptual’ representations from the generative model, and any additional unique sensory details. These unique sensory details can be identified as surprising by the generative model, which captures expectations about the world. This fits with the fact that novelty is thought to promote encoding within the hippocampus [15], and that activity in the hippocampus can reflect prediction error [16].

 The idea that memory is reliant on concepts immediately after encoding explains why memory distortions can emerge rapidly. For example, the Deese-Roediger-McDermott task [2-3] involves showing participants a list of words that are semantically related to a ‘lure word’, which is not present in the list; the lure word is often falsely recalled, even before consolidation, and our simulations display the same effect. Similarly, when participants are told a category before seeing an ambiguous image, their memory is distorted towards that category, and once again our simulations mirror this.

 Generative latent variable models and event construction

 As described above, generative neural networks are often trained by adjusting parameters (loosely corresponding to synaptic connection strengths) to minimise ‘prediction error’ of some kind. These models develop rich representations as a side effect of learning to reconstruct their own inputs or predict the next item in a sequence. Specifically, we use variational autoencoders [17-18], which learn a set of latent variables (i.e. hidden factors behind the data). The network's ‘encoder’ encodes sensory experience as latent variables, while its ‘decoder’ decodes latent variables back to sensory experience. In psychological terms, after training on a class of stimuli these networks can reconstruct stimuli from a partial input, according to the schema for that class, and generate novel stimuli consistent with the schema (or infer novel stimuli by manipulating the latent variables of existing ones).

 Neuroimaging studies suggest that similar neural circuits are involved in recalling and imagining events [19-20]. There is also evidence that damage to the hippocampal formation leads to deficits in imagination, dreaming, and daydreaming [21-23]. Consistent with these findings, we suggest that the hippocampal formation is involved in the ‘decoder’ of a generative latent variable model, i.e. that it helps decode conceptual representations produced by the neocortex into experiences. We are able to simulate how hippocampus and neocortex works together to build conceptual knowledge or ‘schema’ and to construct memories and imagine and infer future situations.

 Summary

 As memories are replayed over the course of systems consolidation, they update a generative, or predictive, model of the world, which supports multiple cognitive abilities. Even right after an experience, remembering involves imagining the past based on concepts, combining some stored details with our expectations about what happened. Imagining and reasoning about potential future situations is also supported by the same system.

 References

1. Bartlett, F. C. Remembering: A Study In Experimental and Social Psychology(Cambridge Univ. Press, 1932).
2. Deese, J. On the prediction of occurrence of particular verbal intrusions in immediate recall.  Exp. Psychol.58, 17–22 (1959).
3. Roediger, H. L. & McDermott, K. B. Creating false memories: remembering words not presented in lists.  Exp. Psychol. Lear. Mem. Cogn.21, 803–814 (1995).
4. Squire, L. R. & Alvarez, P. Retrograde amnesia and memory consolidation: a neurobiological perspective.  Opin. Neurobiol.5, 169–177 (1995).
5. Winocur, G. & Moscovitch, M. Memory transformation and systems consolidation.  Int. Neuropsychol. Soc.17, 766–780 (2011).
6. Marr, D. A theory for cerebral neocortex.  R. Soc. Lond. B176, 161–234 (1970).
7. Marr, D. Simple memory: a theory for archicortex.  Trans. R. Soc. Lond. B262, 23–81 (1971).
8. McClelland, J. L., McNaughton, B. L. & O’Reilly, R. C. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory.  Rev.102, 419–457 (1995).
9. Intraub, H. & Richardson, M. Wide-angle memories of close-up scenes.  Exp. Psychol. Learn. Mem. Cogn.15, 179–187 (1989).
10. Bainbridge, W. A. & Baker, C. I. Boundaries extend and contract in scene memory depending on image properties.  Biol.30, 537–543 (2020).
11. Durrant, S. J., Taylor, C., Cairney, S. & Lewis, P. A. Sleep-dependent consolidation of statistical learning. Neuropsychologia49, 1322–1331 (2011).
12. Richards, B. A. et al. Patterns across multiple memories are identified over time.  Neurosci.17, 981–986 (2014).
13. Ellenbogen, J. M., Hu, P. T., Payne, J. D., Titone, D. & Walker, M. P. Human relational memory requires time and sleep.  Natl Acad. Sci. USA104, 7723–7728 (2007).
14. Quiroga, R. Q. Concept cells: the building blocks of declarative memory functions.  Rev. Neurosci.13, 587–597 (2012).
15. Hasselmo, M. E., Wyble, B. P. & Wallenstein, G. V. Encoding and retrieval of episodic memories: role of cholinergic and GABAergic modulation in the hippocampus. Hippocampus6, 693–708 (1996).
16. Kumaran, D. & Maguire, E. A. An unexpected sequence of events: mismatch detection in the human hippocampus. PLoS Biol.4, e424 (2006).
17. Kingma, D. P. & Welling, M. Auto-encoding variational Bayes. Preprint at https://arxiv.org/abs/1312.6114(2013).
18. Kingma, D. P. & Welling, M. An introduction to variational autoencoders.  Trends Mach. Learn.12, 307–392 (2019).
19. Addis, D. R., Wong, A. T. & Schacter, D. L. Remembering the past and imagining the future: common and distinct neural substrates during event construction and elaboration. Neuropsychologia45, 1363–1377 (2007).
20. Hassabis, D. & Maguire, E. A. Deconstructing episodic memory with construction. Trends Cogn. Sci.11, 299–306 (2007).
21. Hassabis, D., Kumaran, D., Vann, S. D. & Maguire, E. A. Patients with hippocampal amnesia cannot imagine new experiences.  Natl Acad. Sci. USA104, 1726–1731 (2007).
22. Spanó, G. et al. Dreaming with hippocampal damage. Elife9, e56211 (2020).
23. McCormick, C., Rosenthal, C. R., Miller, T. D. & Maguire, E. A. Mind-wandering in people with hippocampal damage.  Neurosci.38, 2745–2754 (2018).