A convolutional neural network (CNN) was implemented to predict the response of LARC patients receiving neoadjuvant chemoradiation therapy. The pre-treatment MRI, and the early-treatment follow-up MRI done at 2-3 weeks after the initiation of radiation were used. The MRI protocol included T2, DWI and DCE. A total of 41 patients were studied, with 8 pCR, 27 Tumor Regression Grade 1, and 9 TRG 2+3. The prediction accuracy was 0.71-0.89 for pCR vs. non-pCR; 0.70-0.77 for TRG(0+1) vs. TRG(2+3), not very good due to the limitations of a relatively small dataset. Using manually extracted tumor features in conjunction with neural network classifiers may achieve a higher accuracy.