Two deep learning methods using the convolutional neural network (CNN) was implemented to segment the rectal cancer in 197 LARC patients, with tumor ROI outlined by a radiologist. For each patient, six frames, including T2, 2 DWI sequences and 3 LAVA sequences, were used for training and validation. The Dice similarity coefficient (DSC) value were used to compare the results of the proposed algorithm and the ROI outlined by reader. The mean DSC was 0.67 and 0.78 from each these 2 method respectively. The proposed algorithms especially the combined serials of U-Net showed improved performance compared to prior published work with individual sequence only. Our work showed the deep-learning with combined image sequence can provide as a promising tool for fully automatic tumor localization and segmentation for rectal cancer.