Meeting Banner
Abstract #3389

Glio-LLaMA-Vision: A Vision-Language Model for Molecular Prediction, Radiology Report Generation, and VQA in Adult-type Diffuse Gliomas

Yae Won Park1, Myeongkyun Kang2, Sang Hyun Park2, and Sung Soo Ahn1
1Yonsei University College of Medicine, Seoul, Korea, Republic of, 2Daegu Gyeongbuk Institute of Science and Technology, Daegu, Korea, Republic of

Synopsis

Keywords: Tumors (Pre-Treatment), Tumors, Glioma; Vision-Language Model; Large Language Model

Motivation: Leveraging a pre-trained large vision-language model may show robust performance for radiology of adult-type diffuse gliomas.

Goal(s): To establish a robust vision-language model for molecular subtyping, radiology report generation, and visual question answering (VQA) in adult-type diffuse gliomas.

Approach: MRI and paired radiology reports from 1,001 adult-type diffuse gliomas patients were included as the institutional training set. A vision-language model, Glio-LLaMA-Vision, was developed from LLaMA 3.1 pre-trained on 2.79 million biomedical text-image pairs and was optimized via fine-tuning from the training set. The performance was validated on external test sets.

Results: Glio-LLaMA-Vision showed robust performance on molecular subtyping, radiology report generation, and VQA.

Impact: Glio-LLaMA-Vision shows promising performance in molecular subtype prediction, radiology report generation, and VQA in adult-type diffuse gliomas. Notably, our current study provides a practical paradigm of adapting general domain LLMs to applications in a specific medical domain.

How to access this content:

For one year after publication, abstracts and videos are only open to registrants of this annual meeting. Registrants should use their existing login information. Non-registrant access can be purchased via the ISMRM E-Library.

After one year, current ISMRM & ISMRT members get free access to both the abstracts and videos. Non-members and non-registrants must purchase access via the ISMRM E-Library.

After two years, the meeting proceedings (abstracts) are opened to the public and require no login information. Videos remain behind password for access by members, registrants and E-Library customers.

Click here for more information on becoming a member.

Keywords