In our new paper we use something called 'reinforcement learning' to better understand psychiatric symptoms that we have known about for close to a century. Surprising as it may sound, aspects of this work were actually pre-empted by the Douglas' Adams classic The Hitchhiker's Guide to the Galaxy. To understand how, we need to first delve into a bit of history...

Freud was among the first researchers to recognise that psychiatric conditions are associated with changes in the way individuals process and use information - what scientists call cognition and behaviour. However, he thought that these changes were buried deep inside the mind and inaccessible without many many sessions of psychoanalysis. Fast forward to the 1960s and researchers such as Aaron Beck disagreed. While they did agree that cognitive and behavioural changes were critical for mental health, they argued that these symptoms were not buried. In fact, a sufferer could actually describe their symptoms. For depression and anxiety, Beck called one key set of symptoms 'negative schemata'. This describes how a depressed individual has the opposite of 'rose tinted spectacles'. They view everything a negative light.

Since these schemata were not deeply buried, researchers developed ways of measuring them by asking patients questions. For instance; how much do you agree with the following statement: "If I don’t set the highest standards for myself, I am likely to end up a second-rate person"? The more schemata you endorse, the more depressed you are. Importantly understanding these schemata led to the development of talking (i.e., psychological) therapies to treat and tackle these schemata. Namely, Cognitive Behavioural Therapy, or simply CBT. Unlike psychoanalysis, CBT recognises that individuals have access to and can describe their symptoms and, moreover, that they can learn and practice strategies to fix them.

CBT is one of the great mental health success stories. Across the United Kingdom, about 50% of anxious or depressed individuals who receive CBT will recover. Given how common these conditions are, this means are able to help a lot of people. 

At the same time, however, this also means we are still failing to help at least half of sufferers. And while some extra people will recover with medication, there still remains about a quarter to a third of sufferers who will not respond to any of our current treatments. Developing a deeper understanding of these symptoms emerge may eventually allow us to close this treatment gap.

In the decades following the initial development of CBT the world changed drastically. The advent of computers meant that we did not need to rely solely on patients telling us about their symptoms we could use computerised tasks to get them to show us their symptoms.

This work has borne out Beck's ideas about negative schemata. Across a wide range of computerised tasks; tasks asking people to remember words, or press buttons when they spot positive words or pictures; we can quantify negative schemata (now often referred to more simply as 'negative bias'). In the 1990s and early 2000s, researchers could now show that depressed and anxious individuals made different responses in terms of 'reaction times' or 'accuracy'; quantitative measures derived from these tasks.

These measures became more and more sophisticated, but at the same time the analysis remained somewhat crude. Human beings might change the way they behave from moment to moment in subtle and interesting ways, but analysis of these tasks generally ignored these subtleties in favour of 'average' performance. This would be akin to asking a patient how their day was 'on average', rather than asking how they responded to a specific scenarios (e.g., how did you feel specifically when you missed your train this morning?). We know intuitively that the devil is in the details - how an individual responds from moment to moment is very important. If we just collapse things into averages we lose these details.

Recognition of the importance of moment-to-moment changes in behaviour brings us to our new paper. 

In a parallel, completely unrelated, research world - the world of computers and maths and robots - mathematical techniques that can asses how learning and behaviour changes from moment to moment have been developed.

One extremely influential approach is 'reinforcement learning'. This approach has been used for all sorts of exciting applications, for instance to develop computerised 'agents' with human level (or better!) performance on computer games. 

So, the question we asked ourselves is this: if we can use this approach to develop fancy agents that behave like 'optimal' humans, can we also use it to understand how and why human behaviour might change when we are unwell?

In other words, we wanted to use reinforcement learning to better understand how anxious and depressed individuals behave from moment to moment. To do this, we first measured the behaviour of group of individuals who were experiencing symptoms of mood and anxiety disorders, as well as a group who were not, on a (somewhat boring!) '4-armed bandit task' (which is a fancy way of saying - they have 4 slot machines and have to decide which of the 4 to play each time).

We then took the maths underlying reinforcement learning and, for each person, we ask what numbers should we plug into the equations to make the reinforcement learning algorithm play the task like these people. In other words, for each real person - those with mood and anxiety symptoms and those without -  we can generate an artificial computerised agents that mimics their behaviour. To those who have read The Hitchhiker's guide to the Galaxy, this may sound strangely familiar!

In The Hitchhiker's guide to the Galaxy, Marvin the 'paranoid android' is a robot who behaves like he is anxious and depressed. What we are doing here is really not a million miles away from this. We are developing 'agents' that behave like real anxious and depressed people. Of course, the joke in the book is, why on earth would someone develop an anxious robot?! Well, one genuine and serious reason is that if we can create agents that mimic anxious or depressed individuals we can ask, what about the equations that we have used to create these agents (reinforcement learning or otherwise) causes this difference? And if we know what causes differences we can ask if it is possible to develop treatments to fix this.

In fact, in our paper, we show that a component of the maths - something called a 'punishment learning rate' - can be tweaked to make an agent demonstrate behaviour that looks rather like an anxious or depressed human on our task. This punishment learning rate controls how quickly behaviour changes after a something bad happens. Specifically, anxious or depressed humans change their behaviour much faster after something bad happens. So, if the slot machine they were playing in our game starts taking their money they will stop choosing it faster. This is a good strategy under certain circumstances, but in others it might not be helpful. To give an extreme example - if you hear about a train crash, does that mean you should learn never to take the train again? Or should you wait to see how rare train crashes are? The former situation would be a prime example of a negative bias, or negative schemata - I should never take a train again because I will be involved in a horrific accident. I have updated my understanding of the risk of train crashes too quickly.

What this means is that we can now use these new fancy computational approaches to explain decades old observations. The difference, however, is that we have moved away from just describing behaviours towards understanding potential mechanisms by which the behaviour is generated on a moment to moment basis. We can now take these observations forward and ask, for example, how does the brain generate these computations? And how do different treatments influence these computations? And can we use this understanding to better target current treatments and develop new treatments?

So, what does the Hitchhiker's Guide to the Galaxy have to do with anxiety and depression symptoms? Well, we can now use computational modelling to generate anxious and depressed 'robots', that demonstrate symptoms we have known about for decades, and which will hopefully provide real insight into how we can improve treatments for anxiety and depression.