Iron is an important source of MRI contrast in the brain. Herein, we investigated the influence of the cellular and subcellular iron distribution on the iron-induced MR contrast. Quantitative MRI on post mortem brain samples was combined with quantitative iron mapping and numerical simulations of local field distributions. We show that iron is heterogeneously distributed in both grey and white matter as well as in subcortical nuclei and different scales of heterogeneity play a role for MR contrast in these regions. Our results provide an important step towards quantitative understanding of iron induced MR-contrast and its microstructural underpinnings.