The idea: a movie’s content matters for naturalistic fMRI brain-behavior mapping

When most people picture a brain scan, they imagine someone lying inside a cylindrical machine, trying their best not to move or fall asleep as the machine gently beeps, thinking about whatever comes to mind. This is resting-state fMRI, and it has been the dominant paradigm for studying how brain regions communicate with each other for years. But what if the person is watching a movie in the scanner? Everyone sees the same frames, gasps at the same plot twist, hears the same dialogue - and yet each person processes and interprets what they see in their own way in the context of their personality, memories, and emotional state. Movie-watching engages perception, attention, emotion, and social cognition all at once, and prior work has shown that the brain connectivity patterns captured during naturalistic paradigms can at times predict demographics, i.e. age and sex, or behavior, i.e. cognitive or personality scores, better than those captured at rest. 

But this raises an important question about naturalistic brain scans. When people watch the same movie, their brains become more synchronized - meaning they respond to the same events at the same moments. Does that increased synchrony reduce the individual variability that we need to perform brain-behavior mapping? Or does a compelling movie actually sharpen those individual signals by boosting the signal-to-noise ratio of fMRI data, notoriously poor in resting-state fMRI? And does every movie do this equally, or do some stimuli unlock richer, more informative brain patterns than others? 

In this work, we look at naturalistic paradigms from a new perspective and set out to investigate just how much the features of the visual or audio stimuli drive brain-behavior mapping accuracy. In the Human Connectome Project's 7T dataset, during fMRI participants watched 13 different movie clips - from Hollywood scenes like Inception and Ocean's Eleven to independent films - alongside resting-state sessions. We found that prediction accuracy varied across clips, with some evoking brain connectivity patterns that predicted cognitive scores well, rivaling or even outperforming the resting-state scans that were five times longer, with others performing more modestly. Once observing that not all movie stimuli increase brain-behavior mapping accuracy in the same way, the next question was: what are the features within a movie that drive brain-behavior accuracy?

Neural synchrony vs SNR: a tradeoff?

Part of the answer lies in what happens when a group of people watches the same movie: their brains start to synchronize. A gripping scene pulls viewers' attention to the same place at the same time, while a boring scene may mean viewers’ attention wanders freely, and this shows up in the fMRI signal as varying similarity in brain activity across people. Interestingly, we found that movies eliciting higher synchrony across the group also tended to produce better cognitive predictions.

This makes intuitive sense: higher brain synchrony likely reflects improved signal quality in the fMRI data, because everyone's brain is time-locked to the same stimulus events. But there's an interesting tradeoff here - too much synchrony could in principle reduce the individual differences that brain-behavior models rely on. Brain-behavior mapping needs differences between people to work - if synchrony pushed everyone's brain into the same activation state, there would be nothing left to distinguish one person from another. So which force wins? In the movie-watching fMRI data we used here, the boost in signal quality won. Even in the most synchronizing clips, each person's brain still carried a distinct signature that was meaningful for behavioral predictions. Synchrony helped us see individual differences of brain circuitry related to cognitive scores more clearly, rather than blurring them away.

We dug deeper to investigate this phenomena by zooming in on moment-to-moment fluctuations within each movie clip using a dynamic functional connectome, sliding-window approach. What we found replicated the main findings at a more granular level - even within a single movie, windows with higher neural synchrony also had the most accurate predictions of cognition.

Different movies, different views of the brain

Clips and windows with social, human interactions and faces were where the peaks in prediction accuracy of cognition occurred. A word cloud of the most predictive moments was dominated by terms like "person," "adult," "talk," and "look." The brain, it seems, is most revealing about cognition when it is engaged with other people, even fictional ones on a screen. Interestingly, salient features appeared to vary by the outcome being predicted. Features driving sex classification were not related to humans, instead they appeared to be the opposite, where the word cloud emphasized terms like “object”, “physical entity” and “artifact”. This implies that different movie features may differently impact brain-behavior mapping accuracy depending on what outcome is being predicted. 

For us, this work prompts some very exciting questions. Can we identify certain movies with specific stimuli that can further boost neural synchrony and prediction accuracy for a specific behavioral target? If and how do brain responses to movies change across the lifespan, or in neurological or psychiatric conditions? And how can we push the deep learning framework further - not just to predict better, but to understand what circuits are being probed by a given movie and driving accuracy?

The brain's inner workings are most transparent when it is engaged with the world, and we're just starting to map this space of engagement.