Detection of small lesions in magnetic resonance (MR) images is currently one of the most challenging tasks. Compared with detection in natural images, existing methods cannot accurately detect small lesions with a limited amount of information in MR images. To solve the problems, we propose a novel multi-task convolutional neural network, which simultaneously regresses the lesion number and detects the lesion location. We train and evaluate the end-to-end network to count and locate extreme small lesions within 3-5 voxels on a mouse brain MR image dataset with point annotations. Our network outperforms other methods on sensitivity and precision.