The high risk of cognitive deficits is a major concern for parents and clinicians caring for premature babies. Early and accurate identification of children at risk is urgently needed for early treatment decision. We propose a deep transfer learning model to predict cognitive deficits at 2 years corrected age using brain structural connectome data obtained at term. The proposed model was able to identify infants at high-risk of later cognitive deficit with an accuracy of 78.5% and an AUC of 0.75. The predicted cognitive scores were significantly correlated with corresponding Bayley-III cognitive scores, with a Pearson’s correlation coefficient of 0.48.