Recent evidence suggests APOE4 is the most potent genetic risk factor for late-onset Alzheimer’s disease. However, researchers have not treated its impact on brain aging. We proposed a 3D Medical Image Global Regional (MIGR) Transformer to determine whether APOE4 affects brain age gap estimate using MRI. We achieved a mean absolute error of 4.71 on brain age estimate in a large healthy dataset (n = 2852) with an age range of 18-97 years from 13 public datasets. We investigated the brainAGE on three stages of Alzheimer’s disease. We found an accelerating trend of brain aging for APOE4 carriers.