Fetal brain imaging is a cornerstone of prenatal screening and early diagnosis of congenital anomalies. Knowledge of fetal gestational age is the key to the accurate assessment of brain development. This study develops an attention-based deep learning model to predict gestational age of fetal brain. The proposed model is an end-to-end framework that combines key insights from multi-view MRI including axial, coronal, and sagittal views. The model uses age-activated weakly-supervised attention maps to enable rotation-invariant localization of fetal brain among background noise. We evaluate our method on a collected fetal brain MRI cohort and achieve promising age prediction performance.