Observation of crosstalk between inflammatory IVD and vertebral Modic changes (MCs) has been reported. This study aims to further evaluate possible interaction between MCs and intervertebral discs by utilizing a variety of MR image contrasts and visually displaying the relation using attention maps. The attention maps displayed that both the MC in the vertebra and the surrounding tissue, mainly the intervertebral discs, are important for the classifier. This indicates that there is possible interaction between MC and the surrounding intervertebral discs. The choice of contrast in the images used to train the CNN affected the distribution of the attention maps.