Multi-parametric magnetic resonance imaging (mpMRI) provides high sensitivity and specificity for breast cancer diagnosis. Accurate breast tumor segmentation in mpMRI can help physicians achieve better clinical managements. Existing deep learning models have presented promising performances. However, the effective exploitation and fusion of information provided in mpMRI still need further investigation. In this study, we propose a convolutional neural network (CNN) with adaptive convolution kernels (AdaCNN) to automatically extract and absorb the useful information from multiple MRI sequences. Extensive experiments are conducted, and the proposed method can generate better breast tumor segmentation results than those obtained by CNNs with normal convolutions.