The development of high-level automated vehicles faces considerable safety challenges nowadays. The major obstacle preventing the mass production of high-level automated vehicles is the safety of the intended functionality accidents caused by functional deficiencies under a triggering condition, especially those related to perception algorithms. Humans are essentially a special kind of sensor, and it is a meaningful thing to study human-in-loop decision-making based on humans’ states for over coming perception algorithm function deficiencies and improving safety of the intended functionality. For high-level automated vehicles, human will serve as the role of passenger, and the mental activities of passenger may be different from diver. By analyse the influence on passengers’ mental activities caused by driving scenario risk via fNIRS, we provides a possible solution to design a human-centred intelligent system to promise safe driving for high-level automated vehicles using passengers’ driving risk cognition.

Experimental Paradigm

This experiment was performed in driving simulator. The participants were requested to sit on a driving simulator and to look at the front scenario. Beside, the participants were required to press the keyboard when they hear a stimulating sound or they feel dangerous, they were not required to do other things. Four kinds of challenging scenarios were used in our experiment: lead vehicle autonomous emergency braking in a short distance, lead vehicle cut-in from left lane in a short distance, lead vehicle cut-in from right lane in a short distance and pedestrian crossing road from right scenarios.

Data Analysis Process

Risk field is a safety indicator that indicates a dangerous degree of driving scenario. The risk field indicator at a moment involves relative longitudinal distance between target vehicle and ego vehicle, relative speed, and those information is related to time to collision (Fig. 1). Firstly, each driving scenario were divided into a low-risk episode and a high-risk episode based on the objective risk field indicator. Specifically, the episode in which the indicator is larger than 0.05 is considered as a high-risk episode, and otherwise it is considered as a low-risk episode.  Secondly, low-risk features and high-risk features of concentration of oxygenated hemoglobinwere extracted by K-SVD dictionary learning and mean values from low-risk and high-risk episodes, respectively; Finally, the T-value analysis method was used to assess the area sensitive to driving scenario risk in the prefrontal contex of passengers based on fNIRS for high-level automated vehicles.

The differences in the T-value based on mean value features of   between low-risk and high-risk episodes are shown in Fig 2, From the T-value result based on mean value features, it is derived that the T-values of lead vehicle autonomous emergency braking in a short distance, lead vehicle cut-in from left lane in a short distance, lead vehicle cut-in from right lane in a short distance and pedestrian crossing road from right in the channel 8 are maximum, compared with other channels.

Besides, we also analyzed the influence on mental activity caused by sex and driving experience,  using boxplots of  in channel 8 about driving experience and sex. The boxplots of  is shown in Fig 3. LN stands for the low-risk episode data about those participants who do not have valid driving experience; HN stands for the high-risk episode data about those participants who do not have valid driving experience; LY stands for the low-risk episode data about those participants who have valid driving experience; HY stands for the high-risk episode data about those participants who have valid driving experience; LM stands for the low-risk episode data about those male participants; HM stands for the high-risk episode data about male participants; LF stands for the low-risk episode data about female participants; HF stands for the high-risk episode data about female participants.  For lead vehicle autonomous emergency braking in a short-distance scenario, the difference of  between low-risk and high-risk episodes for those participants who have valid driving experience are more obvious the difference of those participants who do have valid driving experience; similarly, the difference  of   between low-risk and high-risk episodes caused by male participants that female participants.

Conclusion

In this study, We compared passengers' mental activities in low-risk and high-risk episodes and explored the impact of driving scenario risk on their cognitive processes using fNIRS technology. By compare   between low-risk and high-risk episodes  using the T-value and boxplots, three conclusions were found: 1) the mental activities of passengers caused by driving scenario risk in the Brodmann area 10 are very active; 2) driving experience makes people more sensitive to longitudinal risks; 3) Male participants are more sensitive to longitudinal risks than female participants.

Implications for Safer Driving

The research has important implications for improving safety of the intended functionality. By recognizing and quantifying passengers' mental activities related to driving risk, it becomes possible to implement appropriate measures to enhance overall safety. Additionally, passengers’ mental states can be incorporated as an essential variable in driving safety systems. By continually monitoring passengers’ mental states, systems can provide timely alerts or interventions to mitigate risks and sustain safe driving conditions. Besides, the  of passengers on prefrontal cortex could be combined with reinforcement learning to realize passenger-in-loop decision-making for high-level automated vehicles.

Support

The more details content can be find in the paper of “Driving risk cognition of passengers in highly automated driving based on the prefrontal cortex activity via fNIRS”，(Wang, H., Zhang, X., Li, J. et al. Driving risk cognition of passengers in highly automated driving based on the prefrontal cortex activity via fNIRS. Sci Rep 13, 15839 (2023). https://doi.org/10.1038/s41598-023-41549-9)

The code and supplementary material available at https://github.com/SOTIF-AVLab/fNIRS