Simultaneous PET/MRI is a powerful multimodality imaging technique for both anatomical and functional imaging. Here we propose a novel method for high-quality PET image reconstruction from ultra-low-dose (more than 99% reduction compared to current practice) PET scanning by using multi-contrast-MRI. A multi-scale fully convolutional network was developed for solving the reconstruction. The proposed method is compared with other methods on a Glioblastoma(GBM) clinical dataset. Results show that our method achieves superior image quality compared with state-of-the-art methods in low-dose PET reconstruction. Besides, quantitative and qualitative evaluations indicate that multi-contrast MRI significantly improves the reconstruction quality with better structural details.