We evaluated the short-term repeatability of convolutional neural networks (CNNs) in detecting prostate cancer (PCa) using DWI collected from patients who underwent same day test-retest MRI scans. DWI was post-processed using monoexponential fit (ADCm). Two models with similar architecture were trained on test-retest scans and short-term repeatability of network predictions in terms of intra-class correlation coefficient (ICC(3,1)) was evaluated. Although the observed ICC(3,1) was high for CNN when optimized for classification performance, our results suggest that network optimization with respect to classification performance might not yield the best repeatability. Higher repeatability was observed at lower learning rates.