Abstract #4009

Evaluation of effectiveness and fairness of foundation models in multi-organ segmentation

Qing Li¹, Yizhe Zhang², Yan Li³, Yajing Zhang⁴, Longyu Sun¹, Mengting Sun¹, Qirong Li⁵, Zian Wang⁵, Meng Liu¹, Xumei Hu¹, Shuo Wang⁶, and Chengyan Wang¹

¹Human Phenome Institute, Fudan University, Shanghai, China, ²School of Computer Science and Engineering, Nanjing university of science and technology, Nanjing, China, ³School of Medicine, Shanghai Jiao Tong University, Shanghai, China, ⁴GE Healthcare, Beijing, China, ⁵School of Computer Science, Fudan University, Shanghai, China, ⁶Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai, China

Synopsis

Keywords: AI/ML Software, Machine Learning/Artificial Intelligence, Foundation model, Organ segmentation, Accuracy, Fairness

Motivation: Foundation models have significant potential in organ segmentation. However, They would face challenges regards effectiveness and fairness.

Goal(s): This study aims to evaluate the effectiveness and fairness of foundation models in multi-organ segmentation.

Approach: Foundation models (SAT, SAM, MedSAM, and TotalSegmentator) were included in this study. Bounding box prompts were used with SAM and MedSAM. DICE was used to evaluate segmentation effectiveness. For fairness evaluation, the DICE was grouped by gender, age, and BMI, and ANOVA was used to compute significance between groups.

Results: The effectiveness of SAM and MedSAM is more significant than other mdoels. However, their fairness needs to be improved.

Impact: This study systematically evaluates the variations in segmentation effectiveness of foundation models across different organs and the fairness issues, which finds the shortcomings of the current foundation models and plays an important role in guiding future improvements of foundation models.

How to access this content:

For one year after publication, abstracts and videos are only open to registrants of this annual meeting. Registrants should use their existing login information. Non-registrant access can be purchased via the ISMRM E-Library.

After one year, current ISMRM & ISMRT members get free access to both the abstracts and videos. Non-members and non-registrants must purchase access via the ISMRM E-Library.

After two years, the meeting proceedings (abstracts) are opened to the public and require no login information. Videos remain behind password for access by members, registrants and E-Library customers.

Click here for more information on becoming a member.