Post-acquisition data harmonization promises to unlock multi-site data for deep learning applications. In turn, this rests on measuring image similarity. Here, we investigate the sensitivity of several similarity measures to fourteen acquisition protocols of a 3D T1-weighted (MPRAGE) contrast. Standard similarity metrics, a deep perceptual loss and a segmentation loss are extracted between image pairs and compared. The perceptual loss is highly correlated with L1 distance and outperforms other metrics in detecting acquisition parameter changes. The segmentation loss, however, is poorly correlated with other metrics, suggesting that these image similarity metrics alone aren't sufficient to harmonize data for clinical applications.