A deep learning method using the convolutional neural network (CNN) was implemented to segment rectal cancer in 48 patients. Six sets of images (one T2, Two DWI, three DCE) were used as inputs. The Dice Similarity Coefficient (DSC) was used to evaluate results generated by the CNN algorithm compared to the manually outlined ground truth. When the search was done on the entire image the mean DSC was 0.64, and the errors were mainly from tissues outside the rectum. The rectum could be easily segmented, and when the search was confined within 1.5 times of rectal area, the DSC was improved to 0.75.