Investigating physician trust in AI systems: where is the line between effective collaboration and over-reliance?

AI advice systems need to work alongside doctors to be effective on-the-ground. Here, we study if and how doctors trust the advice coming from these systems, and find a significant risk of over-reliance.
Investigating physician trust in AI systems: where is the line between effective collaboration and over-reliance?

AI and ML advice systems hold a great deal of potential for clinical settings, from early detection of breast cancer to reducing racial disparities in pain treatment. But beyond achieving good performance in isolation, they must also be able to operate in complex socio-technical environments and work effectively with human decision-makers.  

Our paper focuses on how expert and non-expert physicians modulate their trust in AI-generated advice in order to walk a tight line: if physicians do not trust AI advice they will not use it, but blind trust could lead to medical errors. To evaluate this dynamic, we presented radiologists and internal/emergency medicine (IM/EM) physicians with a series of chest x-rays and diagnostic advice. We varied two things: whether we presented the advice as coming from an AI system or a fellow radiologist, and whether the advice was correct or incorrect.  

An image with a flowchart of the study procedure.  On the top half, an icon of a doctor leads to an icon of an X-ray with a box titled "Advice".  The Advice box contains a "SOURCE" field (which can be either AI or human) and an "ACCURACY" field (either accurate or inaccurate).  An arrow points to text reading "Quality evaluation" followed by "Diagnosis".  The bottom half contains a specific example of an X-ray and the corresponding advice.

We found that all physicians were susceptible to incorrect advice, regardless of its source. For high-risk settings like diagnostic decision making, such over-reliance on advice can be dangerous. When physicians ask for advice from colleagues, it is often after their initial review of the case, and entails a back-and-forth discussion. Automated systems may be less easily engaged in dialogue, which could prime physicians to search for, and accept, confirmatory information in place of conducting a thorough and critical evaluation.  

We also found that the over-reliance effect was more pronounced for physicians with less task expertise (IM/EM physicians). This suggests that task expertise is another important consideration in the deployment of clinical decision-aids; there is likely not a “one size fits all” approach to development and design, and understanding specific users’ behavior will be an integral step. 

Finally, we found significant variation in decision-making even within physicians of a particular expertise level. While some variation amongst physicians is natural, variations due to differences in physicians’ trust of support systems could be minimized through additional guidelines, regulations, or training.  

AI is an important potential tool in healthcare, and we highlight research directions that are likely to be critical to their effective deployment: for example, designing effective ways to communicate advice uncertainty, or on-boarding tools for physicians to understand a system’s limitations and calibrate their trust accordingly.

Overall, the fact that physicians were not able to effectively filter inaccurate advice raises both concerns and opportunities for AI-based decision-support systems in clinical settings. While we are not able to regulate the advice that physicians might give one another, we can aim to design AI systems and interfaces to enable more optimal collaboration. 

Please sign in or register for FREE

If you are a registered user on Research Communities by Springer Nature, please sign in

Subscribe to the Topic

Health Care
Life Sciences > Health Sciences > Health Care
  • npj Digital Medicine npj Digital Medicine

    An online open-access journal dedicated to publishing research in all aspects of digital medicine, including the clinical application and implementation of digital and mobile technologies, virtual healthcare, and novel applications of artificial intelligence and informatics.

Related Collections

With collections, you can get published faster and increase your visibility.

Clinical applications of AI in mental health care

This joint venture Collection between npj Mental Health Research and npj Digital Medicine highlights how AI can be safely, ethically, & impactfully utilized to advance our understanding of mental illnesses & improve patient care.

Publishing Model: Open Access

Deadline: Jun 22, 2024