Anatomy contouring is essential in quantifying the dose delivered to the prostate and surrounding anatomy after low-dose-rate prostate brachytherapy. Currently, five anatomical structures including the prostate, rectum, seminal vesicles, external urinary sphincter, and bladder, are contoured manually by a radiation oncologist. In this work, we investigated six convolutional encoder-decoder networks for automatic segmentation of the five organs. Six pretrained convolutional encoders and two loss functions were investigated. This yielded twelve different models for comparison. Results indicated that classification accuracy of convolutional encoders pretrained on the ImageNet dataset positively correlated with semantic segmentation accuracy in prostate MRI.