The Limits and Challenges of Ophthalmic Image Interpretation

Fundus fluorescein angiography (FFA) plays a vital role in routine examination. However, the interpretation of FFA images has relied on physicians' expertise, which is time-consuming and limited by the availability of specialists. While traditional AI-assisted systems have offered some relief in generating medical reports, they fall short of interactive capabilities and comprehensive professional assessments.

Large Language Models (LLM) Enhances Smart Interaction in Ophthalmic Image Interpretation

In response to these challenges, we introduced an innovative system named FFA-GPT—an automated pipeline that combines multi-modal transformers with large language models (LLMs), designed to address the interpretation of FFA images. The system utilizes an image-text alignment module to convert images into professional medical reports, thereby enhancing the efficiency and accuracy of report generation. Additionally, the inclusion of the GPT module (Llama 2) improves the quality of doctor-patient communication by refining interactive question-answering (QA) processes (Figure 1). The system has shown satisfactory performance in automated report generation, showcasing advanced capabilities in language generation and disease identification. Most importantly, the reports and answers generated by FFA-GPT have received positive approval from professional evaluations.

Future Perspectives: Building an Effective Interactive Bridge for Doctor-Patient Communication

Our study demonstrates that the combination of LLMs and multi-modal transformers can enhance the interpretation of ophthalmic images and facilitate interactive exchanges during medical consultations. As this technology continues to advance and optimize, we foresee a more efficient and interactive dialogue environment between doctors and patients, which will not only enhance the quality of service but also increase patient satisfaction. Looking ahead, we expect that this technology will be widely applied in day-to-day clinical practices, thereby improving the eye care experience for patients worldwide.