This study mainly aimed to build a multi-modality radiomics model for predicting lymph node metastasis (LNM) of rectal cancer (RC) preoperatively. 82 RC patients were enrolled in this study. Two, six, four and eleven optimal features were separately selected for venous phase CT, HR-T2WI, DWI and multi-modality by incorporating CT, HR-T2WI and DWI together. Together with MRI-reported LN status feature, clinical features, and the combination of multi-modality and clinical features, in total seven support vector machine (SVM) classification models were respectively built. Incorporated with clinical features, multi-modality radiomics model provided the most robust predictive performance in predicting LNM.