Birthweight is a commonly used indicator of fetal growth weight and has been associated with neuropsychiatric and neurological sequalae. However, little is known about how birth weight impacts the brain in later life. We found positive associations with total brain, grey matter and normal appearing white matter volumes in later life, but not with white matter microstructure or hyperintensities. This relationship is explained by larger head size, rather than by age-associated tissue atrophy, and is furthermore independent of body size. This suggests that larger birthweight is linked to increased brain tissue reserve, but not age-associated brain features.