By Terje B. Holmlund, Chelsea Chandler & Brita Elvevåg

Stories are fundamental to our human experience. They provide an effective way for us to organize information, and being able to remember and retell stories is a core part of the shared cognition that connects us to those around us. Tools for evaluating if we are able to remember and retell new stories can inform clinicians as to the health of our brain. Our paper argues that new speech technology can help us make such evaluations using automation.

To be able to retell a story, many different processes in the brain must happen in an orderly fashion. We must perceive and remember the story, and be able to create organized speech using relevant words in order to retell it. A lot can go wrong. Simple distractions can throw us off, or a lack of interest in the story can compromise the ability to tell what the story was about. In healthcare, we are focused on more sinister causes of poor story recall, such as the degeneration of the brain in dementia or the multifaceted pathology of psychosis. 

Testing recall performance is expensive. Traditionally, the procedure for the clinician is simply to read a story and listen to the retelling. In our study, stories were read out via smart phones and the retellings were captured by these devices in 25 patients with serious mental illness and 79 healthy participants. 

Expert raters listened to the verbal responses and rated them for retelling accuracy on a scale of 1-6. Our goal was to create an automated method of rating recall accuracy that performed as well as experts. We used natural language processing techniques to calculate the similarity between the original story and the transcribed retelling. A computer program counted the number of words in a retelling that were present in the original story. The program also measured similarity between the original story and the transcribed retelling in what is known as a "semantic vector space” to predict retelling accuracy. The predicted scores correlated well with the human ratings. When transcripts were produced using automatic speech recognition we found that even though there were many errors in the transcripts, automated scores still corresponded surprisingly well with the human evaluations.

Technological advances are fun and inspiring, but it is up to us to discover what problems we may solve with our new abilities. Verbal memory assessment is a good example of where recent technological advances in speech technology can help solve a small part of the healthcare puzzle. Algorithms are far from being able to decide for us how to move forward, it is up to us humans to write the next chapter of the story of our healthcare system

Figure: In this example, a story (abbreviated for this figure) was presented via a mobile device about a girl and her balloons at a birthday party. The task for the participant was to retell it as accurately as possible. Humans listened to the responses and rated them for accuracy. We transcribed the response recordings, and the similarity of these transcriptions to the original story was compared using computers. The performance of the automated scoring procedure correlated well with the human ratings.