Convolutional neural network (CNN) has been regarded to be powerful for lesion characterization in clinical practice. However, local deep feature derived from CNN has two main shortcomings for characterization. First, the convolutional operations typically process within a local neighborhood while ignoring the global dependency. Furthermore, it is unstable to small perturbations in images (e.g., noise or artifacts). Therefore, we propose a denoised local fusion and nonlocal deep feature fusion method to alleviate the above two problems. The proposed method is a general module, which can be integrated into any CNN-based architecture for improving performance of lesion characterization in clinical routine.