Smart Clinics in Ophthalmology: Automatic Fundus Fluorescein Angiography Image Interpretation with Large Language Models

FFA-GPT is an automated system for ophthalmic image interpretation that enables both report generation and interactive question answering for fundus fluorescein angiography images.

Share this post

Choose a social network to share with, or copy the shortened URL to share elsewhere

This is a representation of how your post may appear on social media. The actual post will vary between social networks

The Limits and Challenges of Ophthalmic Image Interpretation

Fundus fluorescein angiography (FFA) plays a vital role in routine examination. However, the interpretation of FFA images has relied on physicians' expertise, which is time-consuming and limited by the availability of specialists. While traditional AI-assisted systems have offered some relief in generating medical reports, they fall short of interactive capabilities and comprehensive professional assessments.

Large Language Models (LLM) Enhances Smart Interaction in Ophthalmic Image Interpretation

In response to these challenges, we introduced an innovative system named FFA-GPT—an automated pipeline that combines multi-modal transformers with large language models (LLMs), designed to address the interpretation of FFA images. The system utilizes an image-text alignment module to convert images into professional medical reports, thereby enhancing the efficiency and accuracy of report generation. Additionally, the inclusion of the GPT module (Llama 2) improves the quality of doctor-patient communication by refining interactive question-answering (QA) processes (Figure 1). The system has shown satisfactory performance in automated report generation, showcasing advanced capabilities in language generation and disease identification. Most importantly, the reports and answers generated by FFA-GPT have received positive approval from professional evaluations.

Future Perspectives: Building an Effective Interactive Bridge for Doctor-Patient Communication

Our study demonstrates that the combination of LLMs and multi-modal transformers can enhance the interpretation of ophthalmic images and facilitate interactive exchanges during medical consultations. As this technology continues to advance and optimize, we foresee a more efficient and interactive dialogue environment between doctors and patients, which will not only enhance the quality of service but also increase patient satisfaction. Looking ahead, we expect that this technology will be widely applied in day-to-day clinical practices, thereby improving the eye care experience for patients worldwide.

Figure 1. Schematic diagram of this study. FFA=fundus fluorescein angiography, GPT=generative pre-trained transformer, BLIP=bootstrapping language-image pre-training.

Please sign in or register for FREE

If you are a registered user on Research Communities by Springer Nature, please sign in

Subscribe to the Topic

Artificial Intelligence
Mathematics and Computing > Computer Science > Artificial Intelligence
Life Sciences > Health Sciences > Clinical Medicine > Ophthalmology
Fluorescence Imaging
Life Sciences > Biological Sciences > Biological Techniques > Biological Imaging > Fluorescence Imaging
Health Care Management
Humanities and Social Sciences > Business and Management > Industries > Health Care Management
Medical and Health Technologies
Life Sciences > Health Sciences > Clinical Medicine > Medical and Health Technologies
  • npj Digital Medicine npj Digital Medicine

    An online open-access journal dedicated to publishing research in all aspects of digital medicine, including the clinical application and implementation of digital and mobile technologies, virtual healthcare, and novel applications of artificial intelligence and informatics.

Related Collections

With collections, you can get published faster and increase your visibility.

Clinical applications of AI in mental health care

This joint venture Collection between npj Mental Health Research and npj Digital Medicine highlights how AI can be safely, ethically, & impactfully utilized to advance our understanding of mental illnesses & improve patient care.

Publishing Model: Open Access

Deadline: Jun 22, 2024

Harnessing digital health technologies to tackle climate change and promote human health

This collection invites research on the use of digital health technologies that innovate solutions to improve sustainable health care practice and delivery.

Publishing Model: Open Access

Deadline: Apr 30, 2024