Physical frailty develops with increasing age and is a chronic state of vulnerability that is associated with disability and cognitive decline. Few studies have assessed the underlying structural brain abnormalities of physical frailty. We therefore examined the association between brain volumes on MRI and physical frailty in a group of 214 non-demented elderly participants. Frail participants showed a lower total brain volume and a lower grey matter volume compared to pre-frail and non-frail participants. Furthermore, pre-frail participants showed more cortical infarcts compared to non-frail participants. These brain abnormalities could be the underlying substrate of the physical frailty phenotype.